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Preface 


The  purpose  of  this  study  was  to  determine  if  a  management 
information  system  could  be  developed  to  support  the  performance 
assessment  process.  The  immediate  need  for  such  a  system  is  to  fully 
implement  the  Air  Force  System  Command  initiatives  for  improved  use  of 
past  performance  information  to  support  source  selections.  The  method 
chosen  for  system  development  included  steps  for  problem  definition, 
requirements  analysis,  and  development  design. 

Although  development  of  a  complete  operational  system  was  out  of  the 
scope  of  this  research  effort,  the  baseline  requirements  and  conceptual 
design  have  been  established.  The  baseline  and  conceptual  design  will 
provide  a  firm  foundation  for  further  development  and  implementation  of 
the  system. 

The  core  of  this  study  was  the  requirements  analysis.  Interviews 
were  conducted  with  twenty-nine  military  and  civilian  personnel  who  had 
experienced  a  performance  risk  assessment  first  hand  and  eight 
individuals  who  are  currently  responsible  for  administration  of  the 
Contract  Performance  Assessment  Reporting  System.  I  am  deeply  indebted 
to  these  people  for  their  time  and  patience.  Special  thanks  must  go  to 
Maj  Larry  Allen,  Capt  Dan  Behne,  Lt  Kathleen  Coombs,  Ms  Cynthia  Keefe, 

Ms  Pat  Olson,  Lt  Col  Gary  Poleskey,  and  Mr  David  Smith  for  putting  in 
that  extra  effort  and  providing  encouragement  when  it  was  really  needed. 

I  would  also  like  to  thank  Ms  Susan  Wright  who  pointed  me  in  the 
right  direction  when  this  research  effort  was  just  beginning.  Ms  Wright 
opened  the  doors  necessary  to  get  this  project  under  way. 


I  would  like  to  extend  my  appreciation  to  Ms  Shirley  Sawyer  tor  her 
ass i stance  in  developing'  an  automated  report  torm  for  performance 
assessments.  For  this  and  her  continuing  support  during  times  of  crisis 
I  am  truly  indebted. 

My  deepest  respect  and  appreciation  are  offered  to  Dr.  Charles  Fenno 
who  was  true  partner  in  this  research.  His  quiet  style  of  motivation  and 
encouragement  enabled  me  to  take  this  project  further  than  I  ever  thought 
possible.  Dr  Fonno’s  persistence  on  quality  was  always  present.  Because 
of  his  persistence,  1  can  honestly  say  that  I  feel  good  about  the 
contents  of  this  book  and  how  it  will  represent  me  to  its  readers. 

Finally,  I  wish  to  thank  my  wife  Terry  ;ind  sons  Paul  and  Andrew. 
Their  love  and  patience  kept  me  going.  Without,  their  support  this 
project  would  never  have  been  completed. 


Paul  W.  Thurston  Jr. 
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Abs  tract 

Air  Force  Systems  Command  has  recent,  ly  put  forth  two  major 
initiatives  for  improving  the  performance  evaluation  process  to  support 
source  selection  decisions:  the  Contractor  Performance  Assessment 
Reporting  System  (CPARS),  a  system  for  recording  and  distributing 
contractor  performance  informal  ion;  and  the  Performance  Risk  Assessment 
Group  (PRAG),  a  panel  of  seasoned  acquisition  experts  chosen  for  each 
source  selection  to  provide  a  performance  risk  assessment  for  each 
proposing  contractor. 

CPARS  exists  as  identical  sets  of  contract  assessment  reports 
located  at  each  of  the  product  divisions.  The  files  are  updated  as  new 
reports  are  mailed  by  the  originating  agency  to  each  of  the  other  product 
divisions.  Data  requests  to  supporf  the  PRAG  are  responded  to  with 
copies  of  the  existing  assessment  reports.  There  is  no  automated 
capability  to  extract,  reduce,  or  transform  the  CPARS  data  into 
information  which  can  be  readily  used  by  the  PRAG. 

The  purpose  of  this  study  was  to  determine  if  a  management 
information  system  based  on  the  CPARS  could  b^  developed  to  support  the 
performance  assessment  process. 

A  tailored  approach  to  the  traditional  method  for  software  systems 
development  was  chosen  to  design  an  automated  system  to  support 
performance  risk  assessment.  This  approach  included  steps  for  problem 
de  f  1  n  i  t.  i  on ,  requ  i  remen  t,  s  ana  1  ys  is,  and  deve  1  opmen  t  des  i  gn . 


Requirements  of  the  system  were  determined  through  structured 
interviews  with  PRAG  members  and  (PARS  focal  points.  CPARS  focal  points 
are  individuals  in  each  product  division  who  are  responsible  for  the 
administration  of  the  CPARS  data  base. 

Implementation  of  the  requirements  analysis  step  resulted  in  a 
complete  description  of  the  procedures  used  and  the  information  required 
to  support  the  performance  risk  assessment  process.  The  role  of  the 
CPARS  as  a  primary  source  of  informat  ion  to  support  this  process  was 
described  and  assessed.  The  administrative  procedures  used  to  collect, 
process,  distribute,  and  protect  contractor  performance  assessment 
reports  were  determined. 

The  informat  ion  collected  from  PRAG  members,  CPARS  focal  points,  and 
applicable  regulations  was  synthesized  into  a  concise  statement  of  the 
required  capabilities  for  the  automated  informat  ion  system  to  support 
performance  assessment. 

The  research  process  followed  the  tailored  methodology  through  the 
development,  of  a  conceptual  model  for  the  system.  The  conceptual  model 
developed  describes  the  applications  necessary  to  support  the  required 
functions.  The  conceptual  model  demonstrates  how  the  applications  of  the 
information  system  can  assist  the  user  in  the  various  steps  of  the 
assessment  process. 

Sufficient  time  allowed  for  the  development  of  an  initial  capability 
which  demonstrates  selected  applications  described  in  the  conceptual 
model.  The  software  operates  on  an  IBM  XT  compatible  computer  system 
using  Enable  (Version  2.15).  A  copy  of  the  software  can  be  obtained  from 
the  Director  of  Research,  AFIT/LSC  Wright -Patterson  AFB  OH  45433-06583. 
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DEFINITION  OF  A  MANAGEMENT  INFORMATION  SYSTEM 


TO  SUPPORT  PERFORMANCE  RISK  ASSESSMENT 


I.  Introduction 


Background 

The  primary  objective  of  the  source  selection  process  is  to  choose 
the  offeror  who  can  best  be  expected  to  meet  the  government’s 
requirements  at  the  lowest  possible  cost  (28:3).  .An  important 
consideration  in  this  process  is  the  determination  of  contractor 
effectiveness  through  analysis  of  prior  performance. 

Over  the  past  25  years,  contractor  past  performance  has  been 
evaluated  in  source  selections  as  either  a  specific  criterion,  an 
assessment  criterion,  or  a  general  consideration.  Significant 
deficiencies  have  teen  found  with  each  of  these  approaches  (39;  3) 

Several  attempts  at  data  base  systems  for  tracking  contractor 
performance  have  also  proven  unsuccessful  (31;  35;  3). 

Effective  use  of  contractor  performance  information  was  emphasized 
in  a  1986  report  to  President  Reagan,  as  one  of  the  critical  elements  in 
the  source  selection  process.  Part  V  of  A  Formula  For  Action,  A  Report 
to  the  President  on  Defense  Aequ i s i t i on  by  the  President ’ s  Blue  Ribbon 
Commission  on  Defense  Management ,  April  1986,  calls  for  "increased  use  of 
commercial -style  competition,  emphasizing  quality  and  established 
performance  as  well  as  price"  (46:62).  This  report,  better  known  as  the 
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"Packard  Comm i s s i on"  report,  states  that  government  procurement  should 
mimic  those  practices  which  have  been  highly  successful  in  industry. 


Typically,  an  industrial  company  will  keep  lists  of  qualified 
suppliers  that  have  maintained  historically  high  standards  of 
product  quality  and  reliability.  As  long  as  these  standards  are 
maintained,  industrial  buyers  do  not  require  exhaustive  inspection, 
and  thereby  save  expense  on  both  sides.  Suppliers  are  highly 
motivated  to  get  and  stay  on  lists  of  qualified  suppliers  by 
consistently  exceeding  quality  control  standards.  (46:62-63) 

In  May  1988,  General  Bernard  P.  Randolph,  Commander  of  Air  Force 

Systems  Command,  issued  his  support  of  the  Packard  Commission  findings 

and  described  how  the  Air  Force  acquisition  community  would  implement 

them. 


Until  now,  the  military  didn’t  rely  too  heavily  on  past 
performance  evaluations;  but,  the  "hit  and  run"  approach  doesn’t 
make  sense.  Past  Performance  will  be  a  key  factor  in  our  source 
selections  from  now  on.  This  initiative  implements  a  Packard 
Commission  finding,  uses  a  well-founded  commercial  practice  that 
prime  contractors  employ  in  their  vendor  rating  systems,  and 
recognizes  good  performers.  The  idea  is  to  take  a  systematic  look 
at  contractors’  past  performance  as  a  risk  consideration.  Just  as 
we  assign  technical  risk  to  a  contractor’s  proposal  in  source 
selection,  we’ll  now  assign  performance  risk  based  on  a  company’s 
track  record  on  past  contracts.  (48:2) 

Air  Force  Systems  Command  has  recently  put  forth  two  major 
initiatives  for  improving  the  performance  evaluation  process:  the 
Contractor  Performance  Assessment  Reporting  System  (CPARS),  a 
centralized,  manual  system  for  recording  and  distributing  contractor  past 
and  present  performance  information;  and  the  Performance  Risk  Assessment 
Group  (PRAG),  a  panel  of  seasoned  acquisition  experts  chosen  for  each 
source  selection  to  provide  a  performance  risk  assessment  for  each 
proposing  contractor. 

Air  Force  Systems  Command’s  Contractor  Performance  Assessment 
Reporting  System  (CPARS)  is  the  most  recent  initiative  for  tracking 
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contractor  past  and  present  performance.  Annually  each  program  manager 
is  required  to  prepare  an  assessment  of  the  contractor’s  performance 
based  on  ten  criteria  (product/system  performance,  software  development, 
engineering  des ign/support ,  schedule,  cost  performance,  product 
assurance,  test  and  evaluation,  management  responsiveness,  integrated 
logistics  support,  and  subcontract  management).  These  criteria  represent 
a  determination  of  performance  made  by  the  government  program  manager 
based  on  a  four-level  scale  ranging  from  unsat i sfactory  to  exceptional. 
Each  assessment  must  be  supported  by  objective  evidence  which  is  supplied 
in  the  program  manager’s  narrative  (1:1,6).  A  copy  of  Air  Force  Systems 
Command  Form  125,  "Contractor  Performance  Assessment  Report"  is  presented 
in  Appendix  A. 

Each  evaluation  must  go  through  a  rigorous  process  to  ensure 
objectivity  and  fairness.  The  program  manager  initiates  the  process  by 
probing  the  functional  experts  for  objective  evidence  to  assess  the 
contractor’s  performance.  After  performing  the  initial  assessment,  the 
"preliminary  assessment  report"  is  sent  by  the  program  manager  to  the 
contractor  for  review.  The  contractor’s  response  is  optional.  If 
provided,  the  reply  should  focus  on  the  objective  portion  of  the  program 
managers’  comments.  The  contractor’s  response  should  be  limited  to  the 
space  provided  on  the  assessment  report  plus  one  additional  page.  The 
response  must  be  returned  to  the  program  manager  within  3Q  days  after 
transmittal  (1:2). 

After  reviewing  the  contractor  comments,  the  program  manager  has  the 
opportunity  to  revise  the  initial  assessment.  This  must  be  accomplished 
on  a  new  form  which  will  be  attached  to  the  preliminary  report.  The 
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program  manager  signs  the  assessment  report  and  submits  it  for  review. 

Ttie  reviewing  official,  at  least  one  level  higher  than  the  program 
manager,  examines  the  assessment  to  make  sure  that  it  is  consistent  with 
other  evaluations  for  that  contract,  and  that  all  ratings  are 
substantiated  with  objective  comments  (1:1).  Copies  of  the  assessment 
report  are  sent  by  the  product  division  CPARS  focal  point  to  each  of  the 
other  product  divisions  for  input  into  the  command  wide  data  base.  The 
original  is  kept  by  the  local  CPARS  focal  point  for  the  originating 
product  division  (1:2). 

The  initial  version  of  CPARS  has  been  limited  to  concept  definition, 
full-scale  development,  and  full-rate  production  contracts  costing  five 
million  dollars  or  more.  The  system  collects  assessments  on  projects 
currently  under  contract  with  any  of  the  five  Air  Force  Systems  Command 
product  divisions  (Aeronautical  Systems  Division,  Ballistic  Systems 
Division,  Electronic  Systems  Division,  Munitions  Systems  Division,  and 
Space  Systems  Division)  (1:1).  CPARS,  in  its  current  state,  exists  as 
identical  sets  of  contract  assessment  reports  located  at  each  of  the  five 
product  divisions.  The  files  are  updated  as  new  reports  are  mailed  by 
the  originating  agency  to  each  of  the  other  product  divisions.  Data 
requests  to  support  source  selections  are  responded  to  with  copies  of  the 
existing  contract  assessment  reports.  There  is  no  automated  capability 
to  extract,  reduce,  or  transform  the  CPARS  data  into  information  which 
can  be  readily  used  by  the  PRAG. 

According  to  Air  Force  Systems  Command  Regulation  800-54, 

Acqu i " i t i on  Management ,  Contractor  Performance  Assessment : 

The  sole  purpose  of  the  CPARS  is  to  provide  program  management 

input  for  a  command-wide  performance  data  base  used  in  Air  Force 
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Systems  Command  source  selections.  Performance  assessments  will  be 
used  as  an  aid  in  awarding  contracts  to  contractors  that 
consistently  produce  quality  products  that  conform  to  requirements 
within  contract  schedule  and  cost.  (1:1) 

Ttie  Performance  Risk  Assessment  Group  (PRAG)  uses  information 
derived  from  the  CPARS  data  base  to  provide  a  performance  evaluation  and 
risk  assessment.  Risk  assessment  is  a  confidence  measure  associated  with 
the  contractor’s  ability  to  perform  the  proposed  effort  (2:1).  Hie 
performance  evaluation  and  risk  assessment  will  consider  the  frequency 
and  severity  of  problems,  types  of  corrective  actions  taken  bv  the 
contractor,  and  trends  of  past  and  present  performance.  It  is  not  the 
intent  of  the  performance  risk  assessment  to  be  a  "simple  arithmetic 
function"  of  a  proposing  contractor’s  performance,  but  rather  a 
collection  of  the  most  relevant  information  with  regard  to  that  contract 
(4: 10) . 

The  PRAG  relies  on  several  different  sources  for  information, 
including  CPARS,  to  make  its  risk  assessment.  Each  source  of  information 
can  be  categorized  by  one  of  three  types  of  data  provided:  contract - 
specific,  contractor-plant,  or  contractor  self  assessment,  as  depicted  in 
Figure  1-1. 

Contract -spec if ic  data  provides  information  about  a  single  program. 
The  primary  method  of  collecting  this  data  in  the  past  has  been  through 
questionnaires  or  interviews  with  program  managers  within  Air  Force 
Systems  Command,  the  Air  Force,  or  other  services.  Recently  the  standard 
source  for  information  on  contract  performance  lias  become  the  CPARS. 
However,  the  lack  of  historical  data  stored  in  the  CPARS  has  limited  the 
use  of  the  CPARS  in  the  PRAG  process.  The  CPARS  data  base  contains 
performance  assessments  for  contracts  which  were  ongoing  in  198b  or 
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Figure  1-1.  Information  Sources  For  PRAG 

later.  No  attempt  was  made  to  update  the  data  base  for  prior  years. 

This  has  forced  the  PRAG,  in  the  short  run,  to  continue  its  reliance  on 
questionnaires  with  program  managers  (RM’s)  for  contract -spec i f ic  data. 

It  is  anticipated  that,  as  the  CPARS  matures,  it  will  be  the  primary 
source  for  programmat ic  information  to  support  source  selections. 

Contract-plant  data  is  provided  by  the  contract  administration 
offices.  These  include  Defense  Contract  Administration  Services  (DCAS), 
Air  Force  Plant  Representative  Offices  (AFPRO),  and  Air  Force  Contract 
Management  Division  (AFCMD).  The  contract  administration  offices  provide 
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global  assessments  of  the  contractor’s  internal  operations  and  management 
functions  (3:Sec  11,6).  Reports  are  standard  within  organizations,  but 
are  not  standard  across  the  contract  administration  offices. 

Contractor  self  assessment  is  the  information  provided  by  the 
contractor  in  response  to  the  Request  For  Proposal.  This  information  is 
proposal  specific  and  provides  the  offerors  the  opportunity  to  express 
their  relevant  experience  to  the  type  of  work  involved  in  the  program. 

Genera i  Issue 

CPARS,  in  its  current  state,  exists  as  five  identical  sets  of 
contract  assessment  reports  located  at  each  of  the  product  divisions. 

The  files  are  updated  as  new  reports  are  mailed  by  the  originat ing  agency 
to  each  of  the  other  product  divisions.  Data  requests  to  support  the 
PRAG  are  responded  to  with  copies  of  the  existing  contract  assessment 
reports.  There  is  no  automated  capability  to  extract,  reduce,  or 
transform  the  CPARS  data  into  information  which  can  be  readily  used  by 
the  PRAG. 

Spec i f i c  Prob 1 em 

The  research  reported  in  this  document  had  four  objectives: 

1.  Define  the  information  required  and  procedures  used  by  the 
Performance  Risk  Assessment  Group  (PRAG)  to  institute  the  current  policy 
for  providing  a  risk  assessment  to  support  the  source  selection  decision 
process ; 

2.  Determine  the  extent  to  which  information  derived  from  a  fully 
operational  Contractor  Performance  Assessment  Reporting  System  (CPARS) 
could  support  the  performance  risk  assessment  process; 
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3.  Determine  the  administrative  procedures  used  to  collect, 
process,  distribute,  and  protect  contractor  performance  information  under 
the  CPARS ; 

4.  Establish  the  requirements  baseline  and  conceptual  design  for  an 
automated  information  system,  based  on  the  CPARS,  to  collect,  process, 
protect ,  and  disseminate  contractor  performance  assessments. 

Invest igat ive  Quest  ions 

To  meet  these  objectives,  it  was  necessary  to  answer  the  following 
research  questions. 

1.  What  procedures  are  currently  used  and  what  types  of  information 
are  required  by  all  PRAGs  to  prepare  performance  risk  assessments  on 
proposing  contractors? 

2.  What  portion  of  the  required  information  can  be  derived  from  the 
data  resident  in  CPARS,  and  how  can  the  data  resident  in  CPARS  be 
processed  to  provide  this  information? 

3.  What  are  the  administrative  procedtires  for  collecting, 
processing,  and  distributing  contractor  performance  information  and  how 
do  these  processes  affect  the  archi tecture  for  an  automated  system? 

4.  What  system  capabilities  are  dictated  by  the  information 
requirements  of  PRAGs  and  the  implementation  procedures  of  the  CPARS? 

5.  How  do  the  established  requirements  affect  the  architecture  for 
the  automated  information  system? 
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II.  Literature  Review 

Secondary  source  material  reported  in  this  chapter  describes  the 
current  implementation  of  the  source  selection  process,  establishes  a 
historical  perspective  on  the  use  of  past  performance  information  in 
source  selections,  and  develops  an  approach  to  information  system  design. 
The  first  topic  presented  in  this  chapter  is  an  overview  of  the  source 
selection  process  as  currently  practiced  in  Air  Force  Systems  Command. 

Source  Select  ion 

Def ini t i on  and  Purpose.  The  strictly  controlled  process  of 
evaluation  and  negotiation  for  competitive  award  is  called  source 
select  ion. 

The  primary  objective  of  the  source  selection  process  is  to 
choose  the  source  whose  proposal  has  the  highest  degree  of 
credibility  and  whose  performance  can  be  expected  to  best  meet  the 
government’s  requirements  at  an  affordable  cost.  (28:3) 

The  two  basic  methods  used  to  award  government  contracts  in  other 

than  emergency  circumstances  are  by  sealed  bids  and  by  negotiation.  The 

differences  between  these  methods  are  highl ighted  in  Table  2-1  (4:11-5). 

Sealed  bidding  is  the  preferred  choice  when  all  of  the  following 

criteria  can  be  met:  time  permits  solicitation;  award  is  based  on  lowest 

price;  purchase  is  for  standard  items  or  services;  and  there  is  not  a 

sole  source  situation  (4:11-4).  If  any  one  of  the  last  three  conditions 

dot's  not  exist,  men  the  contract  must  be  awarded  by  method  of 

negot i at  ion. 

Tiie  negotiation  method  is  used  when  required  items  or  services  are 
not  as  determinate  as  those  in  sealed  bidding,  or  when  contracts  are  to 
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Table  2-1 


|  Sealed  Bidding'  Verses  Negotiation  Method 


Sealed  Bids 

Negot i at  ion 

Sol ici tat  ion 

Solicitation 

Advertisement 

Advertisement 

Receive  Bids 

Receive  Proposals 

Examine 

Evaluate 

Negotiate 

Award  to  lowest 
responsible  bidder 

Award  to  offeror  most 
advantageous  to  the 
governmen t 

be  awarded  on  some  justification  other  than  price.  The  basic  steps  in 
the  negotiation  process  remain  the  same  whether  performing  a  one-time 
purchase  of  a  low  cost  item  or  of  high  cost  items  and  services  over  a 
period  of  many  years. 

Every  type  of  contract  let  by  negotiation  involves  the  selection  of 
a  source  for  award.  However,  the  term  "source  selection"  is  normally 
reserved  for  the  more  formal  process  which  involves  the  establishment  of 
a  structure  for  t he  sole  purpose  of  awarding  a  contract  to  one  or  more 
bidders  for  a  specific  service.  In  this  paper,  source  selection  will 
refer  to  the  structured  process  in  which  competing  proposals  are  accepted 
cuid  evaluated  by  a  structure  specifically  established  to  choose  one  or 
more  offerors  for  a  single  purpose'.  As  stated  earlier,  the'  primary 
objective  of  the  source'  selection  process  is  to  choose  the  offeror  who 
provides  the  most  credible  proposal  and  can  best  be  expected  to  meet  the 
gove-rnment  ’  s  requirements  at  an  acceptable  cost. 


2-2 


Policy.  Part  15  of  the  Federa 1  Acqu i s i t i on  Regulat ions  "Contracting 
by  Negotiation,"  provides  the  criteria  for  establishing  source  selection 
procedures.  The  Federal  Acquisition  Regulations  leave  much  discretion  to 
the  acquiring  agency  to  determine  appropriate  evaluation  factors  and 
their  relative  importance. 

The  Federal  Acquisition  Regulations  explicitly  state  that  both  cost 
and  quality  factors  must  be  evaluated  ;r.  every  source  selection.  For 
cost,  evaluation  factors  center  around  cost  realism  and  completeness. 
Quality  evaluation  factors  "may  be  expressed  in  terms  of  technical 
excellence,  management  capability,  personnel  qualification,  prior 
experience,  past  performance,  and  schedule  compliance"  (54:16,925-6). 

More  detailed  guidance  is  provided  by  Department  of  Defense  directives 
and  service  specific  regulations. 

Air  Force  Implement at  ion.  Air  Force  Regulat ion  70- 15 ,  Formal  Source 
Selection  For  Major  Acquisitions  and  Air  Force  Regulation  7Q-3Q, 
Streamlined  Source  Select  ion  Procedures  are  the  primary  guidance  for 
conducting  source  selections  within  the  Air  Force.  Air  Force  Regulation 
70-15  states  that  contract  award  is  based  on  an  integrated  assessment  of 
proposal  contents  and  general  considerations.  Proposal  contents  are 
evaluated  with  respect  to  cost,  spec i f i c  criteria,  and  assessment 
criteria  (29:14).  Spec i f ic  criteria  relate  to  the  characteristics  of  the 
program.  They  can  include  technical,  management,  logistics,  testing  and 
security.  Assessment  criteria  relate  to  the  offerer’s  understanding  of 
the  program  and  inherent  abilities.  Assessment  criteria  can  include 
reasonableness  of  approach,  understanding  of  requi remerit ,  and  compl lance 
with  requ i remeri t s .  The  subjective  assessment  criteria  are  applied  in 


matrix  fashion  to  the  more  objective  specific  criteria  (20:14).  General 
considerations  include  factors  external  to  the  proposal,  such  as  pre- 
award  surveys  and  past  performance.  The  current  version  of  Air  Force 
Regulation  79-15  states  that  past  performance  can  be  evaluated  as  either 
an  assessment  criterion  for  proposal  evaluation,  as  a  general 
consideration,  or  both  (29:14).  Recent  initiatives,  especially  in  the 
past  performance  area  (CP.ARS  and  PRAGs),  have  changed  the  manner  in  which 
information  is  addressed  for  the  decision  process. 

The  current  implementation  is  depicted  in  Figure  2-1  below.  Source 
selection  is  based  on  the  integrated  assessment  of  performance  risk, 
proposal  rating,  proposal  risk,  and  other  general  considerations  as 
applicable.  Specific  criteria  are  organized  according  to  areas,  items, 
and  factors.  The  offeror’s  proposal  is  evaluated  based  on  specific  and 
assessment  criteria.  A  proposal  rat ing  and  risk  are  assigned  for  each 
item  and  area.  Past  performance  is  included  in  the  integrated  assessment 
as  a  performance  risk.  Performance  risk  may  or  may  not  be  assessed 
according  to  the  same  ar^as,  items,  and  factors  as  the  proposal  rating 
and  proposal  risk.  The  assessment  criteria  are  applied  to  each  specific 
criterion. 

The  Source  Selection  Process  :  Planning  and  Sol  ic  i  tat,  ion .  Source 
selection  is  an  integral  part  of  the  acquisition  process.  Selection 
activities  must  be  considered  up-front  in  both  the  procurement  planning 
and  the  requi  rements  document,  a  in  phases  of  the  process  to  ensure  that 
proper  selection  can,  in  fact,  be  made  when  the  time  comes.  Ihe 
acquisition  process  can  be  broken  down  into  seven  different  phases  as 
shown  in  the  Figure  2-2.  As  part  of  that  process,  source  selection 
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Figure  2-1.  Basis  for  Source  Selection  Decision 
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Figure  2-2.  The  Acquisition  Process 

encompasses  the  procurement  planning,  sol icitation,  and  source 
evaluation/selection  phases  of  the  process  and  is  highly  dependent  on  the 
acquisition  strategy  and  clear/complete  requirements  documentation. 

Procurement  Planning.  The  key  people  in  the  procurement 
planning  process  are  the  program  manager  and  the  procurement  contracting 
officer.  Planning  for  source  selection  begins  up  front  with  the 
development  of  the  Acquisition  Strategy  Document.  The  acquisition 
strategy  document  is  the  program  manager’s  guide  to  provide  the  required 
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capability  to  the  user.  This  document  is  program  oriented  and  may 
encompass  many  different  contracts.  According  to  Air  Force  Regulation 
70-15,  the  first  two  steps  of  the  source  selection  process  are 
development  of  the  Acquisition  Plan  and  the  conducting  of  an  Acquisition 
Strategy  Panel  (20:18).  Contracting  strategy,  competition,  decision  for 
source  selection,  recommendation  of  delegation  for  Source  Selection 
Authority  and  type  of  selection  procedures  must  all  be  documented  in  the 
Acquisition  Plan.  For  the  contract ing  officer  and  program  manager, 
source  selection  must  be  a  consideration  from  the  very  beginniiig  of 
procurement  planning. 

Another  major  task  for  the  program  manager  is  to  develop  the  Source 
Selection  Plan.  The  Source  Selection  Plan  is  the  key  to  ensuring  a 
successful  selection  process.  It  documents  every  activity  from 
preparation  of  the  Request  For  Proposal  to  contract  award.  According  to 
Air  Force  Regulation  70-15,  the  Source  Selection  Plan  must  contain  the 
following  sections  (20:7-8): 

1.  Introduction:  This  is  a  description  of  the  system  or  subsystem 
which  is  being  acquired,  and  how  it  satisfies  the  approved  requirement. 
The  introduction  should  be  tied  closely  to  the  Acquisition  Plan. 

2.  Source  Selection  Organization:  The  organization  structure 
chosen  for  the  source  selection  should  be  described.  This  can  take  on 
the  characteristics  of  one  of  two  basic  types  depending  on  whether  formal 
or  streamlined  procedures  are  used.  The  two  organizational  structures 
are  shown  in  the  figure  below.  Names  of  key  members,  if  known,  should  be 
identified  with  their  position  title  or  functional  area.  This  section 
should  include  other  government  organizations  which  will  participate  in 
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Figure  2-3.  Source  Selection  Organization 

the  process,  and  an  estimate  of  the  total  number  of  personnel  involved, 
including  advisors. 

3..  Screening  Criteria:  This  section  must  include  a  description  of 
the  method  for  soliciting  sources  and  ensuring  competition.  Criteria 
must  be  set  up  to  make  sure  that  the  prospective  contractors/contracting 
teams  have  adequate  security  clearances;  and  sufficient  management, 
financial,  technical,  and  manufacturing  capabilities  necessary  to  perform 
the  task  at  hand. 

4.  Evaluation  Procedures:  This  section  describes  the  process  or 
methodology  by  which  the  proposals  will  be  evaluated.  Much  emphasis  is 
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placed  on  the  procedures  used  to  evaluate  cost  and  provide  *  he  best 
estimate  of  the  total  cost  to  the  government. 

5.  Evaluation  Criteria:  This  is  a  description  of  the  types  of 
information  which  the  contractor  will  be  evaluated  by.  It  includes 
specific  criteria  broken  down  to  areas,  items,  factors,  and  subfactors, 
as  applicable;  and  a  description  of  assessment  and  general  criteria.  The 
criteria  will  correspond  to  the  evaluation  factors  listed  in  the 
Acquisition  Plan.  This  section  should  also  state  the  relative  importance 
of  each  criterion  to  the  overall  evaluation. 

6.  Acquisition  Strategy:  This  is  a  summary  of  the  strategy 
associated  with  the  specific  phase  and  contracting  effort  under 
consideration,  rather  than  the  program  view  which  the  section  name  might 
imply.  It  should  identify  contract  type,  incentives,  major  milestones, 
and  special  contract  clauses  intended  for  use.  In  reality  it  is  a 
summary  of  the  information  included  in  the  Acquisition  Plan. 

7.  Schedule  of  Events:  A  list  of  significant  source  selection 
activities  and  the  time  each  is  to  be  completed.  A  chronological  list  of 
39  different  activities  is  included  in  Attachment  1  of  Air  Force 
Regulation  79- 15 . 

The  Source  Selection  Authority  must  approve  the  Source  Selection 
Plan  before  the  solicitation  phase  can  begin. 

So  1 ic i tat  ion.  The  solicitation  phase  begins  with  advertisement 
in  the  Commerce  Bus iness  Daily  by  the  contracting  officer,  and  involves 
three  major  tasks:  development  and  release  of  a  DRAFT  Request  For 
Proposal,  Approval  of  final  Request  For  Proposal  by  the  Solicitation 
Review  Panel,  and  development  of  the  evaluation  standards. 
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The  Request  For  Proposal  is  the  government  document  used  to  solicit 
response  from  industry  to  satisfy  a  particular  requirement.  It  describes 
the  what,  where,  when,  and  how  of  the  government’s  need.  It  is  critical 
that  this  document  be  an  accurate  representation  of  the  government’s 
needs,  because  many  of  its  elements  will  be  drivers  for  the  life  of  the 
contract.  Often,  a  draft  copy  of  the  Request  For  Proposal  will  be  sent 
to  industry  for  initial  comments.  The  additional  effort  of  sending  out  a 
draft  copy  can  pay  off  large  dividends.  Industry  comments  may  point  out 
inconsistencies ,  potential  cost  drivers,  and  unclear  areas  which  could 
cause  misunderstanding.  Prior  to  release,  the  final  Request  For  Proposal 
must  be  approved  by  the  Solicitation  Review  Panel.  The  purpose  of  this 
review  is  to  make  sure  that  the  Request  For  Proposal  is  a  solid 
foundation  for  a  workable  contract. 

Two  sections  of  the  Request  For  Proposal  which  are  very  important  to 
the  actual  evaluation  are  Section  L  -  "Instructions  For  Proposal 
Preparation"  and  Section  M  -  "Evaluation  Factors  for  Award."  These  two 
sections  describe  to  industry  the  type  of  information  the  government  is 
looking  for  (Section  M)  and  the  format  it  wants  it  in  (Section  L) . 

Section  M  provides  the  actual  evaluation-  criteria  which  will  be  used,  and 
the  relative  importance  of  each.  The  offeror  must  address  each 
evaluation  criteria  in  order  to  provide  a  responsive  proposal.  The  only 
way  prospective  contractors  can  be  differentiated  from  each  other  is 
through  the  evaluation  criteria. 

The  evaluat ion  standards  developed  by  the  program  manager  are  the 
actual  gauge  against  which  the  contractor’s  proposals  will  be  measured. 
They  are  measures  of  minimum  acceptable  levels  of  conformity  to  the 
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specified  requirements.  The  standards  can  be  both  qualitative  and 
quantitative  (4:Vll-6).  The  evaluat ion  st  andards  must  correspond  to  the 
evaluation  criteria  listed  in  Section  M  of  the  Request  For  Proposal. 

The  evaluation  standards  are  not  released  to  any  potential  offeror 
nor  any  one  else  who  is  not  directly  involved  in  the  source  selection 
(20:9).  Prior  to  receipt  of  proposals,  the  standards  must  be  approved  by 
the  Source  Selection  Advisory  Council  or  Source  Selection  Evaluation 
Team.  Even  though  it  is  possible  not  to  start  working-  on  the  standards 
until  after  the  Request  For  Proposal  is  released,  this  practice  is  not 
recommended.  It  is  important  t.o  know  that  standards  can  be  developed  to 
measure  the  evaluation  criteria  supplied  to  the  contractor  in  the  Request 
For  Proposal . 

Key  Players .  To  understand  the  evaluation  process,  it  is  important 
to  be  able  to  recognize  the  key  players  in  the  source  selection 
organization  and  what  their  basic  responsibilities  are. 

Source  Selection  Autho. ity.  The  Source  Selection  Authority  is 
the  decision  authority  for  the  selection  of  the  contracting  source.  "The 
Source  Selection  Authority  is  responsible  for  the  proper  and  efficient 
conduct  of  the  entire  source  selection  process  encompassing  proposal 
solicitation,  evaluation,  selection  and  contract  award”  (20:5).  This 
person  approves  the  Source  Selection  Plan,  appoints  the  chairperson  and 
members  of  the  Source  Selection  Advisory  Council,  authorizes  release 
of  material,  and  provides  necessary  guidance  to  the  Source  Selection 
Advisory  Council  and  Source  Selection  Evaluation  Board  (20:5). 

Source  Selection  Advisory  Counc i 1 .  The  Source  Selection 
Advisory  Council  reviews  and  approves  the  evaluation  standards, 


2-11 


determines  appropriate  weighting  of  evaluation  factors,  and  designates 
the  chairperson  and  members  of  the  Source  Selection  Evaluation  Board. 

The  Council  is  responsible  to  review  and  evaluate  the  analysis  performed 
by  the  Evaluation  Board,  compare  competing  offeror’s  proposals,  prepare 
the  Source  Selection  Advisory  Council  Analysis  Report  and  provide 
guidance  as  needed  to  the  Source  Selection  Authority  (20:5). 

Source  Selection  Evaluat ion  Board.  The  Source  Selection 
Evaluation  Board  conducts  a  detailed  review  of  the  proposals  and 
evaluates  the  offeror’s  submission  based  on  the  evaluation  criteria  and 
measured  by  the  evaluation  standards.  It  must  submit  evaluation  reports 
to  the  Source  Selection  Advisory  Council  for  its  analysis  and  be  prepared 
to  provide  guidance  when  necessary.  The  Source  Selection  Evaluation 
Board  wri  1 1  also  establish  a  Contract  Def ini t izat ion  Team  (20:6). 

Source  Selection  Evaluation  Team.  In  a  streaml ined  source 
selection,  the  functions  performed  by  the  Source  Selection  Advisory 
Council  and  Evaluation  Board  are  combined  and  performed  by  the  Source 
Selection  Evaluation  Team.  The  Source  Selection  Evaluation  Team  has  also 
been  referred  to  as  a  Source  Selection  Evaluation  Group  or  a  Proposal 
Evaluation  and  Analysis  Group  (4:V-12).  The  Source  Selection  Evaluation 
Team  is  divided  into  two  teams,  one  responsible  for  the  technical 
evaluation  and  one  responsible  for  the  contractual  and  cost  evaluation 
( 4  :  V- 1 2 ) . 

Performance  Risk  Assessment  Group  ( PRAG ) .  The  PRAG  consists  of 
a  panel  of  seasoned  acquisition  experts  chosen  for  each  source  selection 
to  provide  a  performance  risk  assessment  for  the  proposing  contractors. 
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Performance  risk  is  assessed  at  the  item  and/or  area  level  and  is  briefed 
by  the  PRAG  Chai man  to  the  Source  Selection  Authority. 

Program  Manager /Program  Off  ice.  The  program  office  is 
responsible  to  develop  the  Acquisition  Strategy,  assist  in  preparation  of 
the  Acquisition  Plan,  and  develop  the  Source  Selection  Plan.  The  program 
office  needs  to  develop  the  evaluation  criteria  (or  factors)  and  the 
evaluation  standards.  The  program  office  is  also  responsible  for  all  the 
administrative  details  of  the  source  selection  and  protection  of  the 
sensitive  data. 

Contract ing  Of f icer .  The  contracting  officer  is  responsible 
for  authoring  and  maintaining  the  Acquisition  Plan,  issuing  the 
solicitation,  determining  which  contractors  are  within  the  competitive 
range,  negotiating  definitive  contracts  with  all  offerors  in  the 
competitive  range,  and  conducting  all  correspondence  between  the 
government  and  the  contractor.  The  contracting  officer  is  appointed  the 
leader  of  the  Contract  Def ini t izat ion  Team  (20:6). 

The  Source  Select  ion  Process :  Proposal  Evaluation  and  Select  ion. 

The  process  is  described  in  the  terms  of  the  formal  source  selection 
organ izat ion.  The  primary  difference  between  what  is  described  below  and 
the  streamlined  process  is  that  the  responsibilities  of  both  the  Source 
Selection  Evaluation  Board  and  Advisory  Council  are  combined  and 
performed  by  a  much  leaner  Source  Selection  Evaluation  Team. 

Proposal  Evaluat ion.  The  key  players  in  the  proposal 
evaluation  process  are  the  members  of  the  Source  Selection  Evaluation 
Board  and  their  technical  advisors.  As  stated  earlier,  it  is  the 
responsibility  of  the  Source  Selection  Evaluation  Board  to  perform  the 
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detailed  analysis  of  the  offeror’s  submission.  The  foundation  of  the 
evaluation  process  is  the  source  selection  criteria  as  dnsrr’bed  in 
section  M  of  the  Request  For  Proposal.  The  people  performing  the 
evaluation  (Board  members  and  advisors)  are  organized  according  to  a 
hierarchical  structure  of  .AREAS,  ITEMS,  FACTORS  and  when  necessary,  SUB¬ 
FACTORS.  An  example  of  the  organization  structure  is  presented  in  Figure 
2-4.  Actual  evaluation  criteria  will  differ  depending  on  the 
requirements  of  program. 


Pos i t ion 

Level  Title  Evaluation  Criteria 


Area 

Area  Chief 

1  1 

1  1 

!  Technical  | 

1  I 

t  < 

!  Management  ' 

\  1 

1  i 

;  Cos  t.  ; 

!  (T)  ! 

1  1 

1  1 

!  (M)  ! 

1  1 

1  1 

!  (C)  | 

i  I 

i  i 

Item 

Item  Chief 

1  1 

1  1 

J  Software  Design  J 

1  1 

1  1 

J  Systems  Engineering  J 

!  (T.l)  ! 

1  1 

i  » 

:  (T.2)  i 

I  1 

1  1 

Factor  Evaluation 

1 

1 

Des ign  Methodology  J 

i  i 

i  t 

[  Adherence  to  Schedule  J 

Per sonne 1 

(T.i.  i)  ; 

!  (T.l. 2)  ! 

1  1 

1  > 

Figure  2-4.  Evaluation  Structure 
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Each  factor  (and,  when  applicable,  subfactor)  is  measured  against 
the  evaluation  standards  (which  were  developed  by  the  program 
manager/program  office  and  approved  by  the  Source  Selection  Advisory 
Council).  Factors  and  standards  correspond  to  each  other.  For  example, 
standard  T.1.1  will  correspond  to  "Design  Methodology"  as  shown  in  Figure 
2-4.  A  rating  of  +,  /,  or  -,  indicating  wnether  the  offeror's  proposal 
exceeds,  meets,  or  does  not  meet  minimum  requirements  is  assigned  to  each 
of  the  factors  (20:12).  Narrative  must  be  provided  to  support  each 
rating',  and  should  include  specific  examples  and  references  to  government 
documents  and  the  offeror's  proposal.  Evaluations  are  grouped  at  the 
item  level  and  documented  on  factor  summary  work  sheets. 

The  Item  Chiefs  are  responsible  to  review  the  factor  evaluations  and 
develop  an  item  summary  and  assessment.  The  Item  Chief  must  determine 
how  well  the  offerors  strategy,  organization,  or  proposed  design  meets 
the  specified  requi rements .  The  Item  Chief  describes  the  offeror’s 
proposal,  as  it  applies  to  that  item,  and  lists  both  the  strong  and  weak 
points.  Each  item  is  assigned  a  rating  of  Blue,  Green,  Yellow,  or  Red. 
The  ratings  are  described  in  Table  2-2.  The  assignment  of  ratings  is  a 
subjective  process  based  on  the  objective  fa.:‘or  evolua' ; ons . 

Finally,  the  Item  Chief  assigns  to  each  item  a  proposal  risk  of  Low, 
Moderate,  or  High.  Risk  assessments  are  described  in  Table  2-3.  The 
proposal  risk  assessment  is  a  determination  of  how  likely  the  offeror  is 
to  meet  specified  requirements  given  the  proposal  presented  and  the  level 
of  government  intervention  necessary  to  make  him  succeed.  An  excellent 
or  Blue  proposal  rating  for  a  given  item  could  very  well  have  a  high  risk 
associated  with  it  due  to  the  difficulty  of  what  has  been  presented. 
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Table  2-2 


Table  2-3 


Proposal  Ratings  (20:12)  Risk  Assessment  (20:8) 


Color 

Code 

Performance  Level 

Risk 

Rat ing 

Assessment 

Blue 

Exceptional.  Performance 
exceeds  contractual 
requirements . 

Low 

Not  likely  to  disrupt 
schedule,  increase  cost 
or  degrade  performance. 
Only  normal  government 

Green 

Satisfactory.  No 
problems,  or  only  minor 

monitoring  is  required. 

problems  for  which 
solutions  are  in  hand. 

Moderate 

Has  potential  to  disrupt 
schedule,  increase  cost 
and  degrade  performance. 

Yel low 

Marginal.  Problems  for 
which  some  question 
exists  as  to  whether 
solut  ion  is  adequate, 

However,  difficulties 
can  be  overcome  with 
close  monitoring. 

but  resolution  appears 
to  be  within  the 
contractor’s  ability. 

High 

Likely  to  disrupt 
schedule,  increase  cost 
and  degrade  performance 
even  with  close 

Red 

Unsat  is factory.  There 
is  serious  problem  whicn 

government  monitoring. 

may  be  outside  the 
contractor’s  ability  to 
solve. 

N/A 

Not  /Applicable. 

Similarly  a  Marginal  or  Yellow  item  rating  given  because  the  offeror 
barely  meets  or  is  just  below  specified  requirements,  could  have  a  low 
risk  assessment  because  what  the  offeror  is  attempting  is  simple  and  has 
been  accompl ished  before. 

A  similar  process  is  conducted  by  the  .Area  Chiefs  to  combine  the 
individual  item  summaries  into  an  area  summary.  The  Area  Chiefs  look  at 
the  strengths  and  weaknesses,  ratings  and  risks  at  the  item  level,  and 
the  proposal  descriptions  in  order  to  assign  proposal  ratings  and  risk 
assessments . 
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Often  an  offeror’s  proposal  will  contain  wording  which  is  difficult 
to  comprehend,  or  sections  which  are  not  well  explained  or  supported. 

For  these  cases  an  individual  can  write  a  clarification  request  (CR)  to 
allow  the  offeror  a  chance  to  clear  up  the  understanding.  In  instances 
where  the  offeror’s  proposal  is  unacceptable,  a  deficiency  report  (DR) 
cart  also  be  written.  Issuing  deficiency  reports  to  the  offerors  provides 
them  the  opportunity  to  correct  their  deficiencies.  Because  all 
communication  must  flow  through  the  contracting  officer,  clarification 
requests  and  deficiency  reports  cannot  be  sent  directly  from  the 
initiator  to  the  offeror.  The  specific  process  in  which  clarification 
requests  and  deficiency  reports  flow  to  the  offeror  is  depicted  below. 

In  this  process,  duplicate  reports  and  requests  are  eliminated  and  only 
the  most  critical  are  sent  forward. 


Ini t  iate 

CR/DR  ->  Item 

Ch  i  e  f  -  >  Area 

Review  Chief  ->  CR/DR 

Review  Review  ->  Contracting 

Board  Officer  ->  Offeror 


Figure  2-5.  Clarification  Request/Deficiency  Report  Process 

Based  on  the  evaluation  by  the  Source  Selection  Evaluation  Board, 
guidance  by  the  Contract  Def ini t izat ion  Team,  and  approval  by  the  Source 
Selection  Advisory  Council,  the  Cont ract ing  Of f icer  determines  which 
offerors  are  in  the  "Competitive  Range”  (i.e.,  those  offerors  which  have 
a  reasonable  chance  of  being  selected)  (20:12).  Written  (clarifications 
requests  and  deficiency  reports)  and  oral  discussions  are  conducted  only 
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with  contracu/rs  which  are  in  the  competitive  range.  Those  offerors 
which  are  not  in  the  competitive  range  are  promptly  notified  by  the 
Contracting  Officer.  Discussions  are  held  with  all  remaining  offerors 
through  the  Contract  Def init izat ion  Team  and  the  Contracting  Officer. 

The  Source  Selection  Authority  has  the  option  to  award  the  contract 
without  discussions  if  a  clear  winner  can  be  seen. 

Comparison.  Up  until  this  point,  the  offerors’  proposals  have 
been  compared  only  to  the  evaluation  standards  which  were  set  up  prior  to 
receipt  of  the  proposals.  Selection  procedures  begin  with  the  comparison 
of  the  offerors  by  the  Source  Selection  Advisory  Comic! 1.  The  comparison 
of  proposals  is  tased  on  the  analysis  of  the  evaluations  prepared  by  the 
Source  Selection  Evaluation  Board  (20:13).  The  process  involves  looking 
across  proposals  to  ensure  ratings  have  been  used  consistently.  Very 
often  a  +  or  -  is  added  to  the  color  score  to  indicate  the  high  and  low- 
range.  Significant  differences  between  proposals  should  be  highlighted. 
Results  of  this  process  are  documented  in  the  Source  Selection  Advisory- 
Council  Analysis  Report  (20:13). 

Performance  Risk  Assessment .  The  Performance  Risk  Assessment 
Group  reviews  relevant  contract  performance  information  (contractor 
performance  assessment  reports  or  questionnaires),  pre-award  surveys,  and 
the  offeror’s  past  performance  proposal  in  order  to  make  an  assessment  of 
the  risk  for  selecting  the  offeror  based  on  its  track  record.  A  very- 
detailed  description  of  this  process  will  be  presented  in  Chapter  IV. 

Selection .  The  Source  Selection  Advisory  Council  analysis, 
supported  by  the  Source  Selection  Evaluation  Board  evaluation,  is 
transformed  into  a  standard  briefing  format  and  presented  to  the  Source 
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Selection  Authority  for  decision.  The  proposal  evaluation  for  each 
contractor  and  assessment  of  the  proposal  risk  are  presented  to  the 
Source  Selection  Authority  by  the  Source  Selection  Advisory  Council 
Chairperson.  The  evaluation  and  proposal  risk  are  presented  at  the  area 
level  and  supported  by  the  corresponding  items.  The  PRAG  Chairperson 
presents  a  risk  assessment  of  the  offerors’  past  performance  co  the 
Source  Selection  Authority  using  the  same  areas  and  items  as  the  Source 
Selection  Advisory  Council  briefing. 

Based  on  the  proposal  rating,  proposal  risk  assessment,  performance 
risk  assessment,  and  any  applicable  general  considerations,  the  Source 
Selection  Authority  will  make  an  integrated  assessment  of  the  offerors’ 
capability  to  perform  the  task  at  hand.  If  the  process  has  been 
successful,  the  Source  Selection  Authority  will  select  the  source  "whose 
proposal  has  the  highest  degree  of  credibility  and  whose  performance  can 
be  expected  to  best  meet  the  government’s  requirements  at  an  affordable 
cost"  (20:3) . 

The  emphasis  on  past  performance  information  to  assist  the  Source 
Selection  Authority  has  varied  over  the  last  25  years.  The  following 
section  presents  a  historical  perspective  on  the  use  and  collection  of 
past  performance  information  to  support  source  selections. 

Historical  Use  of  Performance  Information 

The  CPARS  is  not  the  first  attempt  to  develop  a  data  base  of  past 
performance  informat  ion.  In  1961  President  Kennedy  tasked  David  Bell, 
then  Director  of  the  Bureau  of  the  Budget,  to  conduct  a  study  of  the 
government’s  expertise  on  Research  and  Development  contracts  (31:3).  The 
"Bell  Report,"  30  April  1962,  raised  the  possibility  of  establishing  a 
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"central  and  fairly  formal  means  of  reporting  methods  and  experiences  and 
recording-  these  permanently"  (31:3,4).  The  large  dollar-value 
development  and  production  program,  more  commonly  referred  to  as  the 
Contractor  Performance  Evaluation  Program,  was  initiated  by  Department  of 
Defense  Directive  5176.38  on  5  October  1964.  The  Contractor  Performance 
Evaluation  Program  collected  performance  information  on  all  Department  of 
Defense  programs  with  projected  costs  of  two  million  dollars  or  more. 

The  data  base  consisted  of  semiannual  reports  created  by  the  government 
program  managers  (35:4-6).  The  development  process  and  format  of  the 
reports  were  very  similar  to  those  used  today  for  the  CPARS.  The  central 
file  for  the  Contractor  Performance  Evaluation  Program  was  located  at  the 
Defense  Documentation  Center  (35:15). 

An  October  1968  report,  conducted  by  the  Logistics  Management 
Institute  for  the  Assistant  Secretary  of  Defense  for  Installations  and 
Logistics,  described  three  fundamental  problems  with  the  Contractor 
Performance  Evaluation  Program: 

1.  There  were  too  few  reports  to  provide  a  credible  description  of 
a  contractor’s  performance  (35:19); 

2.  A  lack  of  summarizing  and  organizing  caused  reports  that  could 
be  generated  to  be  too  cumbersome  to  provide  the  person  evaluating  them 
with  the  necessary  information  (35:12); 

3.  There  were  problems  associated  with  ensuring  objectivity  of  the 
program  manager  evaluations  (35:15). 

In  1979,  less  than  six  years  after  the  Contractor  Performance 
Evaluation  Program’s  inception,  the  Assistant  Secretary  of  Defense  for 
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Installations  and  Logistics  requested  an  evaluation  of  the  program,  and 
on  21  December  197G  released  a  letter  officially  cancelling  it  (31:8). 

In  July  1978,  Air  Force  Systems  Command  initiated  a  test  to  evaluate 
the  use  of  past  performance  information  in  source  selections.  Fourteen 
programs  were  selected  from  four  product  divisions  (Armament  Division, 
Aeronautical  Systems  Division,  Electronic  Systems  Division,  and  Space 
Division).  Past  performance  was  rated  and  scored  as  a  specific  criterion 
in  half  of  the  programs  and  used  as  a  general  consideration  in  the  other 
(3: Sec  I I 1,3).  Three  conclusions  were  drawn  from  this  test: 

1.  Neither  method,  as  a  specific  criterion  or  general 
consideration,  was  considered  to  be  satisfactory  for  evaluating 
performance.  The  recommended  use  of  past  performance  was  as  an 
assessment  criterion  relating  to  the  offeror’s  understanding  of  the 
program  and  inherent  abilities  (3:See  III, 3).  The  subjective  assessment 
criterion  could  be  applied  in  matrix  fashion  to  the  more  objective 
specific  criteria. 

2.  Air  Force  Systems  Command  should  establish  procedures  to 
validate  information  provided  by  the  proposing  contractor,  handle 
situations  when  there  is  no  relevant  past  performance,  and  resolve 

di sagreements  between  offeror  and  government  information  (3:Sec  III, 3). 

3.  Contractor  performance  evaluation  should  be  limited  to  "relevant 
performance”  which  must  be  specified  at  the  time  of  solicitation 

( 3 : Sec  111,3). 

In  1979  Air  Force  Systems  Command  issued  policy  changes  which 
required  that  past  performance  information  be  used  as  both  a  general 
consideration  and  an  assessment  criterion.  Relevant  performance  was 
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defined  to  include,  but  not  be  limited  to,  comparable  work  for  the  same 
or  similar  item,  in  the  same  or  similar  acquisition  phase,  performed  by 
the  same  company  or  division/profit  center.  Direction  was  also  provided 
that  an  offeror  not  be  penalized  for  lack  of  recorded  performance,  that 
disagreements  be  settled  during  negotiations,  and  that  information 
provided  by  the  contractor  be  checked  with  the  appropriate  government 
point  of  contact  (3:Sec  III, 4). 

In  1981,  as  a  result  of  a  protest  lodged  against  the  government  for 
considering  past  performance  as  both  a  general  consideration  and  an 
assessment  criterion,  Air  Force  Systems  Command  issued  a  letter  change  to 
Air  Force  Systems  Command  Regulation  80-15  discontinuing  the  use  of  past 
performance  information  as  an  assessment  criterion  and  mandating  its  use 
as  a  general  consideration  (39:viii). 

As  described  previously,  the  current  Air  Force  direction,  Air  Force 
Regulation  70-15,  dated  22  February  1984,  states  that  past  performance 
can  be  used  as  an  assessment  criterion,  a  general  consideration,  or  both. 

According  to  research  performed  by  Major  Jeffery  Norton  in  July- 
1986,  these  contradictions  in  direction  have  led  to  inconsistent  use  of 
performance  information  by  the  Air  Force  Systems  Command  product 
divisions  (39:  ix).  Based  on  interviews  with  the  source  selection 
officers  at  the  five  product  divisions,  Major  Norton  concluded: 

Specific  confusion  was  found  in  the  interpretation  of  the 
applicable  regulations.  .  .  .  The  validity  of  past  performance  as 
an  assessment  criterion  appeared  to  be  directly  affected  by  the 
relevancy  and  recency  of  previous  contractor  effort  and  indirectly 
by  the  fear  of  protest/ l i t igat i on.  All  five  source  selection 
officers  were  aware  of  one  or  more  of  the  previous  protest/ 
litigation  problems  associated  with  application  of  past  performance 
as  a  hard  decision  criterion.  If  there  is  questionable  relevancy  or 
questionable  data  supporting  a  less-than-sat isfactory  assessment, 
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there  is  a  tendency  to  avoid  making  hard  source  selection  decisions 
based  on  past  performance.  (39:x) 

In  1982  Air  Force  Contract  Management  Division  initiated  a  program 
to  collect,  document,  and  report  information  on  programs  exceeding  $100 
million  or  designated  as  a  major  system  acquisition.  The  program  was 
initially  designed  to  collect  information  from  the  Air  Force  Systems 
Command  system  program  offices,  Defense  Logistics  Agency,  contractors, 
and  other  services  on  ten  functional  areas.  This  information  could  then 
be  provided  to  aid  source  selections.  Support  was  never  garnered  from 
the  other  agencies,  arid  the  concept  was  tested  using  Air  Force  Contract 
Management  Division  input  only.  The  program  was  cancelled  in  1984 
because  the  information  it  collected  was  unnecessary  and  costly  when 
compared  to  other  sources,  and  it  did  not  provide  the  information  needed 
by  the  source  selection  authority  to  make  his  decision  (3:Sec  III, 4). 

Electronic  Systems  Division  established  its  own  program  for 
collecting  and  documenting  performance  information.  The  Contract 
Performance  Evaluation  Program  described  in  Electronic  Systems  Division 
Regulation  178-1  collects  periodic  reports  using  the  Air  Force  Systems 
Command  Form  64.  This  program  was  recently  reduced  in  scope,  and  is  now- 
used  to  collect  information  only  on  programs  not  included  in  the  CPARS 
(11). 

For  the  Contract  Performance  Assessment  Reporting  System  to  be 
successful,  it  must  overcome  problems  associated  with  previous  systems. 

It  must  provide  a  credible  description  of  a  contractor’s  performance  in  a 
manner  which  is  not  cumbersome  to  the  Performance  Risk  Assessment  Groups, 
the  program  managers,  or  the  CPARS  administrators. 
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Hie  last  section  in  this  chapter  describes  the  potential  use  of  a 
management  information  system  as  an  automated  form  of  CPARS  to  meet  the 
past  performance  objectives  set  by  Air  Force  Systems  Conmand. 

Information  Systems  Design 

Management  Information  Systems .  An  automated  system  based  on  the 
CPARS  to  support  the  risk  assessment  process  can  be  included  in  the 
general  category  of  management  information  systems.  A  management 
information  system  represents  an  orderly  method  for  collecting  data,  and 
processing'  and  disseminating  past,  present,  and  predictive  information 
relative  to  the  internal  operations  and  external  environment  of  an 
organization  (14).  The  management  information  system  is  used  to  support 
the  operations,  analysis,  planning,  and  decision  making  functions  in  an 
organization  (18:6).  The  potential  use  of  the  management  information 
system  in  an  organization  can  be  described  as  a  hierarchy  of  functions 
ranging  from  transaction  processing  to  strategic  planning.  Gordon  B. 
Davis  and  Margrethe  H.  Olson’s  Management  Information  Systems :  Conceptual 
Foundat ions ,  Structure,  and  Development  uses  a  pyramid  structure  to 
describe  the  four  levels  of  use  of  a  management  information  system 
(Figure  2-6) . 

Each  level  in  the  hierarchy  can  use  information  generated  at  the 
lower  levels  and  has  the  capability  to  introduce  new  information  from 
external  sources.  The  CPARS  as  currently  planned  represents  the  bottom 
three  layers  of  the  management  information  system  pyramid. 

Transaction  processing  is  characteristic  of  the  data  administration 
tasks  of  the  CPARS  focal  point.  It  includes  data  input,  maintenance, 
report,  generation,  and  protection  of  information. 
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.  Strategic 
.  planning  and 
.  decision  making  . 


Management  planning 
and  decision  making 


Operational  planning, 
decision  making  and  control 


Transaction  processing,  inquiries 
and  data  administration 


Figure  2-6.  Management  Information  System  Pyramid 

Operations  planning  and  control  represents  the  interaction  between 
the  government  arid  industry  program  managers.  The  data  acquisition  tasks 
associated  with  the  program  manager’s  evaluation,  the  contractor’s 
response,  program  manager’s  update,  and  verification/validation  by  the 
reviewing  official  are  representative  of  tasks  which  could  be  supported 
by  a  management  information  system. 

Because  of  the  tactical  nature  of  the  PRAG’s  tasks,  providing  a 
performance  risk  assessment  on  potential  contractors  fits  into  the 
management  planning  and  decision  making  tier. 

There  are  no  current  plans  for  use  of  CPARS  information  to  support 
strategic  planning  and  decision  making.  Air  Force  Systems  Conmand 
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Regulation  8QQ-54  specifically  prohibits  use  of  CPARS  information  except 
to  support  source  selections  (1:1).  However,  use  of  the  CPARS  as  an 
essential  part  of  the  Air  Force  Acquisition  Information  Systun  might 
provide  the  necessary  information  to  support  trend  analysis  of  various 
corpora <•  > and  nrovid0  insight'  in  the  long  range  productivity  of  the 
Air  Force/ industrial  complex. 

An  important  distinction  to  make  is  the  difference  between  data  and 
information.  Data  is  defined  as  unstructured  elements  which  have  been 
acquired  through  direct  observation,  experimentation,  or  questioning; 
while  information  is  data  which  has  been  transformed  so  that  it  has 
meaning  and  usefulness  to  the  recipient  (14).  However,  this  difference 
is  relative  to  the  recipient  of  the  information.  That  is,  information  at 
one  level  of  the  pyramid  may  be  considered  unstructured  and  overwhelming 
data  at  a  higher  level.  The  use  of  a  management  information  system 
provides  the  structure  necessary  to  perform  the  transformation  from  data 
to  information  for  the  recipient. 

The  management  information  system  pyramid  and  the  relative  nature  of 
differences  between  information  and  data  can  provide  insight  into  the 
operations  of  CPARS  and  its  various  uses.  The  institution  of  the  CPARS 
had  two  major  objectives:  provide  feedback,  during  the  processing  of  the 
assessment  report,  from  the  government,  program  manager  to  the  industrial 
manager  on  specific  contract  performance;  and  provide  performance 
informat  ion  across  many  contracts  to  assist  the  PRAG  in  developing 
performance  risk  assessments  during  source  selections.  Each  of  these 
objectives  supports  the  goal  to  motivate  the  contractor  to  consistently 
deliver  quality  in  order  to  ensure  future  business.  The  assessment 
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report  for  a  given  contract  represents  information  to  the  government  and 
industrial  managers  for  that  effort.  The  assessment  report  tells  them 
exactly  where  they  stand  for  that  period  of  time.  However,  the 
individual  assessment  report,  or  more  than  likely,  a  stack  of  assessment 
c  m«v  an  overload  unprocessed  data  to  the  DRAG.  Tn 

tact,  this  was  one  of  the  biggest  problems  with  the  now  defunct 
Department  of  Defense  Contractor  Performance  Evaluation  Program  which,  as 
discussed  earlier,  did  not  provide  information  in  a  format  useful  to  the 
recipient . 

Approach  to  Sys  t  em  Deve 1 opment .  The  choice  for  system  development 
strategy  is  critical  to  the  acceptable  completion  of  the  project.  The 
strategy  must,  consider  uncertainty  associated  with  users’  stat  ing  and 
analysts'  extracting  the  information  requirements:  uncertainty  of  the 
requirements’  validity;  proper  implementation  of  the  development /design 
of  the  system;  and  confidence  that  the  system  meets  the  users’  needs. 
According  t  o  Da v i s  and  Olson, 

Development  should  encompass  a  process  to  accommodate 
requirements  changes  and  provide  assurance  that  the  application  is 
an  accurate  and  complete  reflection  of  user  requirements.  (18:564) 

One  strategy  which  attempts  to  satisfy  these  problems  is  the 

traditional  method  for  a  software  systems  development.  It  is  a 

systematic  process  involving  the  five  stages  shown  in  Table  2-4. 

The  completion  of  each  stage  is  marked  by  the  acceptance  of  the  user 

that  the  system  (as  defined  at  that  point)  meets  the  operational 

requirements.  A  firm  set  of  req\urements  must  be  established  prior  to 

entering  the  Development /Design  stage. 
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Table  2-4 


Traditional  Systems  Development  Approach  (27) 


I.  Problem  Definition 

-  Set  Objectives 

-  Detemni^e  Feasibility 

II.  Requirements  .Analysis 

-  Determination 

-  Documentation 

-  Alternatives 

-  Establish  Baseline 

III.  Deve  1  opmen t  /Des  i  gn 

-  Conceptual  Design 

-  Physical  Design 

-  App  1  i  cat.  i ons  Deve  1  opmen t 


IV.  Implementation 

-  Testing 

-  Conversion 

-  Operations 

-  Maintenance 

V.  Evaluation 

-  Cos t /Performance  And i t 

-  Upgrade/Rep I aoement 


Acquirements  .-Ana lysis.  Determining  user  requirements  is  the  most 
difficult  aspect  of  information  system  development.  Davis  and  Olson 
describe  several  factors  which  affect  the  analyst's  ability  to  determine 
ari  accurate  set  of  requirements.  These  include  natural  const r  nts  on 
humans  to  process  information  and  solve  problems  (limits  of  short-term 
memory,  bias  caused  by  recency  or  traditional  ways  of  doing  business,  and 
inability  to  define  the  problem  space),  and  complexity  of  the  information 
requi rements  (18:474-9). 

Four  strategies  which  can  be  used  to  determine  users’  information 
requirements  are  described  he  low  (18:480-8).  These  strategies  can  be 
used  singularly  or  in  combination  with  each  other. 

1.  Asking  users  directly.  Users  are  interviewed  by  the  analyst  to 
determine  the  system’s  requi rements .  One  or  more  of  several  methods  can 
be  used  to  carry  out  the  process.  These  include  closed  and  open 
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questions,  brainstorming',  and  group  consensus.  The  basic  assumption 
underlying'  this  process  is  that  users  can  overcome  the  natural 
constraints  described  above  (18:481). 

2.  Deriving  requirements  from  an  existing  system.  The  system  can 
be  a  replacement  tor  another,  or  a  like  system  may  exist  vhich  can  be 
used  as  the  re< [U i remen t s  base  (18:482). 

3.  Determine  requirements  through  object  system  analysis. 

Informal  ion  systems  produce  output  to  support  some  otner  object  system  or 
function.  By  analyzing  the  applications  of  the  object  system,  the 
information  system’s  requirements  can  be  defined.  Davis  and  Olson 
describe  eight  different  methods  for  determining  requirements  in  this 
manner  (18:482-8). 

4.  Experimentation  with  an  evolving  information  system.  Customary 
procedures  demand  that  a  firm  set  of  requirements  exist  before  the  start 
of  system  development.  Tn  the  experimentation  strategy,  an  initial  set 
of  requirements  is  extracted  from  the  user  and  a  prototype  system  is 
developed.  Tine  prototype  system  is  then  used  as  a  basis  for  the  user  and 
analyst  to  further  refine  requirements  (18:488). 

Chap t or  Summary 

This  chapter  provided  an  overview  of  the  sourer*  selection  process  as 
currently  practiced  in  Air  Force  Systems  Command,  established  a 
historical  perspective  on  the  us°  and  collection  of  past  performance 
informal  ion  to  support  source  selections,  shoved  the  potential  use  of  a 
management  information  system  as  an  automated  form  of  (PARS  to  meet  the 
past  performance  objectives  set  by  Air  Force  Systems  Command,  and 
developed  an  approach  to  information  system  design. 
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III.  Methodology 

As  stated  in  Chapter  I,  the  research  reported  in  this  document  had 
four  primary  objectives: 

Objective  1.  Define  the  information  required  and  procedures  used  by 
the  Performance  Risk  Assessment  Group  (PR-\G)  to  institute  the  current 
policy  for  providing'  a  risk  assessment  to  support  the  source  selection 
dec i s i on  process ; 

Objective  2.  Determine  the  extent  to  which  infoimat ion  derived  from 
a  fully  operational  Contractor  Performance  Assessment  Reporting  System 
(CPARS)  can  support  the  performance  risk  assessment  process; 

Objective  3-  Determine  the  administrative  procedures  used  to 
collect,  process,  distribute,  and  protect  contractor  performance 
information  under  the  CPARS;  and, 

Objective  4.  Establish  the  re<]ui remen ts  baseline  and  conceptual 
desigt 2  for  an  automated  in  format  ion  system,  based  on  the  CPARS,  to 
collect,  process,  protect,  at,d  disseminate  contractor  performance 
asses smen t s . 

A  tailored  approach  to  the  traditional  method  for  software  systems 
development  was  chosen  to  design  an  automated  information  system  to 
assist  the  contractor  assessment  and  performance  risk  assessment 
processes.  The  methodology  used  for  this  research  followed  the  first 
three  steps  of  the  five  step  process  depicted  in  Figure  3-1.  Development 
and  design  was  completed  through  the  conceptual  design  of  the  system. 


Step  1.  Problem  Definition  Step  3.  Development /Design 

Define  Scope  -  Conceptual  Design 

Set  Objectives  -  Physical  Design 

Step  2.  Requirements  Analysis  Step  4.  Imp lementat ion 

Determine  Requirements  Step  5.  Verification/Validation 

Establish  Baseline 


Figure  3-1.  Tailored  Approach  to  Systems  Development 
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Primary  data  was  collected  through  structured  interviews  with 
selected  experts  on  performance  risk  assessment.,  source  select  ion 
procedures,  and  CPARS  administration.  Two  specific  populations  were 
ident i f ied : 

1.  Performance  Risk  Assessment  Groups.  Individuals  in  the 
acquisition  business  who  have  actively  participated  in  the  formal  risk 
assessment  for  contractors  in  a  source  selection.  Two  representatives 
were  identified  for  each  PRAG  which  had  looked  at  (PARS  as  a  source  of 
informat  ion. 

2.  CPARS  Focal  Points.  Individuals  in  each  product  division, 
usually  assigned  to  the  source  selection  office,  who  are  responsible  for 
the  administration  of  the  CPARS  data  base. 

S tep  1_.  Prob  1  em  Def  ini  t  ion 

A  complete  understanding  of  the  acquisition  and  source  select  ion 
processes  was  required  before  this  project  could  be  undertaken, 
knowledge  of  the  historical  use  and  collection  techniques  provided  the 
opportunity  to  identify  similarities  and  differences  between  prior  uses 
of  contractor  performance  information  and  the  current  concepts  of 
performance  risk  assessment.  This  investigation  also  provided  the 
opportunity  to  look  at  problems/successes  of  previous  attempts  to 
centralize  contractor  performance  data. 

This  research  began  with  an  investigation  of  the  Federal,  Department 
of  Defense ,  and  Air  Force  regulations  governing  use  of  past  perf ormance 
information  to  support,  the  source  selection  process.  Air  Force  Systems 
Command’s  Project  STAR  (Strategies  and  Techniques  for  Acquisition 
Responsiveness)  Report,  and  other  technical  papers  were  examined  to 
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establish  the  historical  use  of  past  performance  information.  A  complete 
description  of  source  selection  procedures,  the  Contractor  Performance 
Assessment  Reporting  System,  and  the  Performance  Risk  Assessment  Group 
was  accomplished  through  analysis  of  Air  Force  System  Command  regula t ions 
and  discussions  with  experts  in  the  acquisition  field. 

Step  2 .  Requirements  .Ana lysis 

Requirements  Analysis  was  the  second  step  in  the  five-step  system 
development  methodology.  It  was  in  this  stage  of  the  research  process 
that  a  valid  set  of  requirements  was  determined.  The  compilation  of 
requirements  through  interviews  and  investigation  of  regulations 
fulfilled  the  first  three  objectives  of  this  research  project.  The  needs 
described  by  the  various  sources  were  synthesized  into  a  baseline  of 
system  level  requi remen t.s .  The  development  of  the  "requirements 
baseline"  completed  the  first  part  of  the  final  objective  of  this 
research  project. 

Perform ance  Risk  Assessment  Process.  Investigative  Question  1 

focused  on  describing  the  procedures  and  information  requirements 

associated  with  the  current  imp lementat, ion  of  PRAGs .  The  purpose  of 

Investigation  Question  2  was  to  determine  the  effect  the  CPARS  has  on  the 

performance  risk  assessment  process. 

Investigative  Question  1.  What  procedures  are  currently  used  and 
what  types  of  information  are  required  by  ail  PRAGs  to  prepare 
performance  risk  assessments  on  proposing  contractors? 

Invest igat i ve  Quest  ion  2.  What  portion  of  the  required  information 
can  be  derived  from  the  data  resident  in  CPARS,  and  how  can  the  data 
resident  in  CPARS  be  processed  to  provide  this  information? 

Structured  interviews  with  rcpresent.at.  i  ves  from  PRAGs  identified  by 

the  product  divisions  w  -e  conducted  to  answer  the  first  two 
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investigative  questions.  Each  product  division  provided  names  of  senior 
individuals  (PRAC  chairman  or  other  member)  who  actively  participated  in 
the  formal  risk  assessment  and  junior  individuals  (PRAG  assistant  or 
secretariat)  responsible  for  the  actual  collection  and  transformation  of 
the  performance  data. 

Two  representat ives  were  identified  for  each  PRAG  which  had 
completed  source  selection  under  the  new  format  and  had  looked  at  CPARS 
as  a  source  of  information.  A  complete  description  of  the  PRAGs 
investigated  and  the  specific  responsibilities  of  PRAG  representatives 
interviewed  is  presented  in  Appendix  B:  Descript  ion  of  PRAGs  and  PRAG 
Representat ives .  Appendix  B  also  provides  insight  into  the  differences 
across  the  five  product  divisions. 

A  mix  of  closed  and  open  ended  questions  was  used  to  collect 
information  from  PR\G  representatives  (Appendix  C:  Survey  Instruments ) . 
The  questions  were  designed  to  avoid  yes/no  type  answers  and  to  encourage 
the  persons  being  interviewed  to  share  their  experiences  freely  and  to 
the  fullest  extent  possible.  Primary  data  collected  during  these 
interviews  was  compiled  and  entered  into  a  text  data  base  to  facilitate 
documentation  and  grouping  of  like  requirements. 

Performance  Assessment  Report ing  Process .  The  purpose  of 
Investigative  Question  3  was  to  determine  the  administrative  procedures 
which  are  currently  used  to  collect,  process,  and  distribute  contractor 
performance  information. 

Investigative  Question  3.  What  Eire  the  administrative  procedures 
for  collecting,  processing,  and  distributing  contractor  performance 
inPormat ion. 


c~ructured  interviews  were  conducted  with  the  CPARS  focal  point  for 

each  of  the  five  product  divisions  to  determine  the  administrative 

procedures  and  constraints  involved  in  the  current  performance  assessment 

reporting  process.  A  mix  of  closed  and  open  ended  questions  was  used 

(.Appendix  C:  Survey  Instruments ) .  The  questions  were  designed  to 

encourage  the  persons  being  interviewed  to  describe  the  detailed  steps 

and  the  peculiarities  of  the  process,  and  to  share  their  experiences 

freely  and  to  the  fullest  extent  possible.  Primary  data  collected  during 

these  interviews  was  compiled  and  entered  into  a  data  base  to  facilitate 

documentation  and  grouping  of  like  requirements. 

Requirements  Basel ine.  The  purpose  for  Investigative  Question  4  was 

to  establish  the  requirements  baseline  for  an  automated  information 

system,  based  on  the  CPARS,  to  collect,  process,  protect,  and  disseminate 

contractor  performance  assessments. 

Investigative  Question  4.  What  system  capabilities  are  dictated  by 
the  information  requirements  of  PRAGs  and  the  implementation 
procedures  of  the  CPARS? 

The  requirements  baseline  represents  the  system  level  capabilities 
which  must  be  provided  in  order  to  meet  the  user’s  needs.  The 
administrative  procedures  and  the  information  and  analysis  requirements 
determined  through  interviews  with  the  PRAG  representatives  and  the  CPARS 
focal  points  were  synthesized  into  seven  different  activities  describing 
the  performance  assessment  process:  data  collection,  data  input, 
maintenance,  analysis,  output,  reporting,  and  protection. 


Step  3 .  Deve 1 opment /Des ign 

During  the  development /des ign  stage  of  the  process,  a  conceptual  and 
physical  design  for  the  system  must  be  established.  The  conceptual 
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design  represents  the  system  as  a  set  of  interactions  and  data  flows.  It 
is  a  user  orientation  of  the  system  design.  The  physical  design  maps  the 
capabilities  described  in  the  conceptual  design  to  the  specific  hardware 
and  software  components  of  the  system. 

Conceptual  Design.  The  purpose  of  Investigative  Question  5  was  to 
develop  the  conceptual  model  of  the  automated  system.  The  development  of 
the  conceptual  design  fulfilled  the  final  objective  of  this  research 
project  . 

Investigative  Question  5.  How  do  the  established  requirements 

affect  the  architecture  for  the  automated  information  system? 

To  an.  wer  this  question  the  requirements  organized  by  the  seven 
activities  of  the  system  had  to  be  described  in  terms  of  user 
applications  of  the  automated  system.  Based  on  a  review  of  software 
engineering  principles,  an  appropriate  model  was  chosen  to  describe  the 
different  applications  of  the  system.  Concentration  was  placed  on 
describing  the  interactions  among  the  various  functions  of  the  system. 
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IV.  Results  Of  Requirements  Analysis 

Chapter  IV  describes  the  implementation  of  the  second  step  in  the 
system  development  process  and  the  fulfillment  of  objectives  1  through  4 
of  this  project.  For  clarity,  each  objective  and  investigative  question 
has  been  repeated  just  prior  to  the  section  to  which  it  pertains. 

Requirements  Analysis  is  the  second  step  in  the  five-step  system 
development  methodology.  This  is  the  stage  of  the  research  process  where 
a  valid  set  of  system  level  requirements  was  established.  Requirements 
for  the  system  were  determined  from  three  primary  sources:  existing 
regulations,  interviews  with  PRAG  representatives,  and  interviews  with 
CPARS  Focal  Points. 

The  first  part  of  this  chapter  reports  the  information  obtained  from 
the  interviews  with  the  PRAG  representatives  and  the  CPARS  Focal  Points. 
The  final  portion  of  this  chapter  describes  the  accumulation  of 
requirements  from  all  three  sources  into  a  requirements  baseline. 

Analysis  of  Interviews  w  rh  PRAG  Representatives 

Interviews  with  PRAG  members  were  used  to  describe  the  current 

implementation  of  the  performance  risk  assessment  process.  A  summary  of 

the  results  for  each  quest  ion  is  presented  below.  The  actual  questions 

have  been  included  in  Appendix  C:  Survey  Instruments . 

Objective  1.  Define  the  information  required  and  procedures  used  by 
the  Per  formatter  Risk  Assessment  Group  (PRAG)  to  institute  the  current 
policy  for  providing'  a  risk  assessment  to  support  the  source  selection 
decision  process; 

Investigative  Question  1.  What  procedures  are  currently  used  and 
what  types  of  information  are  required  by  all  PRAGs  to  prepare 
performance  risk  assessments  on  proposing  contractors? 
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Performance  Risk  Assessment  Process .  This  section  provides  an 
overview  of  the  actual  steps  the  PRAGs  went  through  during  the 
performance  risk  assessment  process.  Representatives  were  asked  to 
discuss  the  events  which  took  place  from  the  time  they  were  notified  of 
their  participation  through  the  final  briefing  to  the  Source  Selection 
Authority. 

Although  many  aspects  of  the  process  differed  across  PRAGs  and 
product  divisions,  all  PRAGs  seemed  to  follow-  the  same  seven  steps:  plan, 
obtain  data,  support  the  competitive  range  briefing,  compile  and  evaluate 
information,  assess  risk,  identify  concerns  to  the  procurement 
contracting  officer  for  discussions  with  the  contractor,  and  brief  the 
Source  Selection  Authority. 

Planning.  Four  parts  to  the  planning  step  were  identified: 
establish  responsibilities  and  methodology;  provide  instructions  to 
offerors;  develop  the  questionnaire;  and  establish  relevancy  criteria. 

Establ ish  Respons ibi 1  it ies  and  Methodology.  The  very 
first  step  identified  by  the  PRAG  members  was  to  gather  all  the 
information  that  was  available  (CPARS  regulation,  Air  Force  Regulation 
70-30,  product  division  supplements,  other  pertinent  regulations,  and 
experiences  from  other  PRAGs)  to  find  out  what  a  PRAG  did  and  what  the 
responsibilities  of  the  PRAG  were  (5;  10,  33;  34;  49;  52).  PRAGs  would 
then  set  up  a  methodology  to  identify  what  data  needed  to  be  collected 
and  how;  to  identify  criteria  to  determine  what  data  would  be  useful;  and 
to  describe  how  the  data  would  be  analyzed  (5;  6;  7;  34:  ^8;  49;  52). 

Another  task  identified  was  to  make  sure  the  correct  words  were  in 
the  Request  For  Proposal,  and  look  at  the  source  selection  plan  to  make 
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sure  performance  risk  has  been  addressed.  PRAG  members  stated  that  it  is 
absolutely  critical  that  the  Request  For  Proposal  and  the  Source 
Selection  Plan  track  meticulously  since  these  documents  contain  the 
evaluation  criteria  for  the  entire  source  selection  (5;  38). 

Provide  Instruct  ions  to  Offerors .  At  all  product 
divisions,  as  part  of  the  Request  For  Proposal,  offerors  are  informed 
that  they  are  to  be  evaluated  on  their  past  performance  and  (at 
Aeronautical  and  Electronic  Systems  Divisions)  are  told  the  specific 
criteria  which  would  be  looked  at  (5;  7;  18;  15;  23;  24;  38;  44).  The 
offeror  provides,  as  part  of  the  proposal,  the  contracts  that  demonstrate 
their  capability,  including  any  experience  by  major  subcontractors.  The 
offeror  must  submit  a  volume  titled  "Past  Performance"  citing  information 
on  all  the  programs  they  had  worked  on  during  the  last  three  (and  for 
scm''  programs  vp  to  five)  year a  (5;  7;  18;  15;  16;  23;  24;  29;  32;  38; 

44;  45;  49;  57).  At  Aeronautical  Systems  Division  this  volume  had  to  be 
submitted  two  weeks  prior  to  due  date  of  proposals  in  order  to  allow  a 
head  start  for  mailing  questionnaires  (7;  18;  15;  23;  24;  44). 

The  "Past  Performance"  volume  should  include  for  each  example  of 
relevant  experience,  the 

contract  number  (5;  7;  18;  15;  16;  23;  24;  29;  32;  38;  44;  45; 

49;  57); 

-  acquisition  agency  (5;  38;  45); 

-  program  name  (5;  7;  18;  15;  16;  23;  24;  29;  32;  38;  44;  45;  49; 
57); 

-  name/address /phone  number  of  the  contractor  officer  and  program 
manager  (5;  7;  18;  15;  16;  23;  24;  29;  32;  38;  44;  45;  49;  57); 

-  contract  administrat ion  office  and  the  administrative  contracting 
officer’s  address  and  phone  number  (5;  38;  45); 
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original  contract  cost,  ultimate  contract  cost,  and  reasons  for 
deviation  (5;  7;  18;  15;  23;  24;  38;  44;  45); 

-  original  contract  schedule,  ultimate  contract  schedule,  and 
reasons  for  deviation  (5;  7;  18;  15;  23;  24;  38;  44;  45);  and 

-  a  description  of  the  project  and  how  each  contract  is  a  relevant 
example  of  the  offeror’s  ability  to  perform  against  the  criteria 
stated  in  the  Request  For  Proposal  (5;  7;  18;  15;  23;  24;  38;  44 
45)  . 

The  only  difference  among  the  product  divisions  in  the  information 
requested  was  the  number  of  relevant  examples  requested:  Space  Systems 
Division-  5  examples  (52),  Electronic  Systems  Division  -  18  to  15 
examples  (5),  and  Aeronautical  Systems  Division,  which  did  not  provide 
the  offeror  a  limit,  but  expected  anywhere  between  15  and  68  (44). 

Deve 1 op  Ques t i onna i re .  The  purpose  of  the  quest  ionnaires 
developed  by  the  product  divisions  was  to  get  information  on  contracts 
which  were  not  represented  in  the  CPARS  (44).  Questionnaires  developed 
took  several  different  forms. 

At  Aeronautical,  Ballistic  and  Electronic  Systems  Divisions, 
many  questionnaires  focused  on  the  criteria  specified  in  the  Request  For 
Proposal  (5;  7;  18;  12;  13;  15;  44;  56).  For  one  PRAG  at  Aeronautical 
Systems  Division,  a  more  generalized  questionnaire  was  created  in  an 
deliberate  attempt  to  get  better  answers  (24).  In  all  cases,  the 
questionnaires  were  more  specific  than  the  assessment  reports  (5;  7;  18; 
12;  13;  15;  24;  36;  44;  57).  At  Space  Systems  Division  and  for  one  PRAG 
at  Electronic  Systems  Division,  the  actual  CPARS  form  was  used  as  a 
questionnaire  (42;  45;  52).  For  Munitions  Systems  Division,  the  CPARS 
format  was  used  to  create  a  checklist  of  questions  (29;  49;  57). 

Establ ish  Relevancy  Criteria.  Relevancy  Criteria  were 
used  for  two  purposes:  to  screen  examples  of  performance  before 
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questionnaires  were  sent  to  make  sure  they  were  applicable  to  the 
solicitation,  and  to  determine  the  impact  of  specific  examples  of  past 
performance  when  formulating  the  risk  assessment. 

The  following  criteria  were  identified  by  the  PRAG  members: 
Division/profit  center  (6;  36) 

Product  Division  (37) 

Dollar  thresholds  (5;  25;  33;  38) 

Type  of  contract  (37;  52) 

Nature  of  the  program  (6;  12;  25;  32;  33;  36) 

Technical  description  (5;  33;  38) 

End  items  (52) 

Phase  of  acquisition  cycle  (52) 

Obtain  Data.  The  second  step  identified  by  the  PRAG 
representatives  was  to  obtain  the  necessary  data  to  support  the  risk 
assessment.  The  primary  sources  for  information  were  the  past 
per  forma..?"  volume  provided  by  the  contractor,  the  CPARS,  interviews  or 
surveys  wdth  government  points  of  contact  for  each  program,  and 
information  provided  by  contract  administration  personnel.  The  "Past 
Performance"  volume  was  reviewed  to  see  what  contracts  were  offered  as 
examples  of  relevant  performance  by  the  offer or  (7;  10;  15;  24;  28;  37; 
42;  49;  52).  Contracts  identified  by  the  offerors  were  screened  so  that 
only  the  most  applicable  would  be  looked  at  (32;  42). 

Nearly  every  PRAG  member  stated  that  the  CPARS  was  checked  for 
reports  which  were  relevant  to  the  program  in  source  selection  (5;  6;  7; 
10;  12;  15;  16;  23;  24;  28;  29;  32;  33;  36;  37;  42;  44;  45;  49;  52;  56; 
57).  The  fact  that  the  assessment  reports  are  written  by  program  mangers 
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vho  are  on  the  program  at  the  time,  and  that  the  reports  must  be  shown  to 
the  contractor  makes  the  CPARS  a  very  credible  source  of  information  (5). 

In  only  one  case  did  a  PRAG  have  enough  assessment  reports  to  not 
need  to  augment  them  with  additional  surveys  (33).  In  all  other  cases 
PRAG  members  would  conduct  interviews  or  distribute  questionnaires  to 
other  government  agencies.  PRAGs  contacted  program  managers  and 
procurement  contract  officers  for  each  contract  identified  by  the  offeror 
and  any  other  organizations  that  they  may  know  from  prior  experiences. 

In  many  cases,  questionnaires  were  mailed  to  the  government  point  of 
contact  (6;  7;  10;  12;  13;  15;  24;  25;  36;  37;  44;  51;  56).  Several 
different  methods  were  used  to  help  ensure  questionnaires  were  returned 
quickly  with  a  quality  response.  Questionnaires  were  sent  out  at  the 
director  level  in  order  to  get  commitment  from  management  to  help  ensure 
a  quality  review  (23).  Before  the  questionnaire  is  sent,  the 
organization  would  be  caileu  to  establish  a  point  of  contact  as  high  up 
in  the  chain  as  possible  to  make  sure  that  the  right  person  is  identified 
and  that  the  job  is  given  the  right  priority  (5;  38).  To  speed  up  the 
process,  facsimile  machines  were  use  to  send  the  questionnaire  out  and 
to  receive  the  completed  forms  (5;  38). 

Often  a  tight  source  selection  schedule  would  not  allow  time  to  send 
out  written  questionnaires,  and  PRAGs  chose  to  conduct  interviews  by 
telephone  instead  (16;  28;  29;  32;  42;  45;  49;  52;  57).  Because  the 
interview  was  done  on  the  telephone,  it  turned  out  that  it  was  difficult 
to  validate  the  things  that  were  being  said  <29;  49).  To  increase  the 
"validity"  of  the  telephone  interview,  one  PRAG  used  the  CPARS  form  as  a 
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questionnaire,  had  the  form  typed,  and  then  used  a  facsimile  machine  to 
send  it  to  the  government  point  of  contact  for  signature  (45). 

Interviews/questionnaires  were  also  used  to  get  an  independent 
evaluation  from  administrative  contracting  officers  (45;  57)  and  other 
representatives  from  the  Defense  Contract  Administration  Service  (19;  13; 
51)  and  Air  Force  Plant  Representative  Office  for  each  contractor  (56). 
One  PRAG  elected  to  travel  to  the  contractor  locations  and  meet  with  thQ 
government  plant  representatives  (49;  57)  The  PRAG  talked  not  only  to 
administrative  contracting  officers,  out  also  to  the  quality  assurance, 
configuration,  and  manufacturing  people  (49;  57). 

PRAG  representatives  stated  that  obtaining  data  through  interviews 
and  questionnaires  was  the  most  difficult  oart  of  the  performance  risk 
assessment  process.  Problems  identified  included  finding  the  right 
address /person  to  send  the  questionnaire  to  (24),  waiting  for  the  surveys 
to  come  back,  contacting  government  representatives  (19;  19;  24;  28;  44; 
57),  hesitation  of  people  to  discuss  contractor  performance  over  the 
phone  (32),  inability  to  tell  whether  the  person  being  interviewed  was 
being  totally  objective  (29)  and  asking  questions  in  such  a  way  so  that 
the  other  services  can  understand  what  the  PRAG  was  looking  for  (19). 

Other  sources  of  information  were  the  Defense  Logistics  Agency 
Contractor  Alert  Lists  (28;  37;  45;  49;  52;  57),  Defense  Contract 
Administration  Service  Annual  Reviews  (7;  19),  Air  Force  Plant 
Representative  Office  Contractor  Operation  Reports  (5;  34;  42)  and 
Preaward  Surveys  (7;  16;  24;  28;  44;  57). 

Compet i t ive  Range  Briefing.  At  some  point  prior  to  the  end  of 
the  assessment  process  and  normally  before  all  the  data  is  even 
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collected,  the  competitive  range  must  be  briefed  to  the  Source  Selection 
Authority.  The  competitive  range  is  the  determination  by  the  procurement 
contracting  officer  of  the  offerors  which  have  a  reasonable  chance  of 
being  selected.  Discussions  are  conducted  only  with  contractors  in  the 
competitive  range,  and  the  Source  Selection  Authority  has  the  option  to 
award  the  contract  without  discussions  if  a  clear  winner  can  be  seen. 

Two  strategies  have  been  used  by  the  PRAG  to  support  the  competitive 
range  briefing.  One  PRAG  chose  to  brief  only  the  methodology  chosen  to 
implement  the  performance  risk  assessment  process  and  not  to  present  any 
assessment  information  (49).  Other  PRAGs  attempted  to  gather  as  much 
data  as  possible  and  perform  an  initial  assessment  based  on  the  offeror’s 
proposal  and  the  available  assessment  reports  (10;  33;  37;  42;  52).  Many 
PRAG  members  did  not  even  mention  the  competitive  range  briefing  as  a 
step  in  the  process. 

Comp i 1 e  and  Evaluate  Informat  ion.  Once  collected,  the  data  had 
to  be  compiled  and  processed  in  order  for  the  PRAG  to  develop  the 
performance  risk  assessment.  The  implementation  of  this  step  differed 
greatly  across  the  product  divisions. 

Aeronautical  Systems  Division.  At  Aeronaut ical  Systems 
Division,  the  secretariat  (PRAG  assistant)  was  responsible  to  build  a  set 
of  books  for  the  PRAG  members.  The  organization  of  these  books  evolved 
over  several  different  PRAGs.  The  first  attempt  divided  data  up  by  item, 
a  process  which  involved  a  lot  of  copying,  cutting,  and  pasting.  Each 
book  had  item  specific  information  from  the  offeror,  followed  by  the 
government  data  for  each  relevant  contract  to  that  item  (15).  Lieutenant 
Kathleen  Coombs,  the  PRAG  secretariat  for  that  program,  stated  that  it 
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was  a  very  t ime- intens ive  process  (15).  There  could  be  as  many  as  sixty 
examples  per  contract.or  or  as  few  as  fifteen  (7;  15;  19;  24;  44). 

The  process  evolved  into  a  system  of  three  books  with  separate 
sections  for  offeror  and  government  supplied  data  (15).  The  first  book, 
which  was  the  offeror’s  Past  Performance  Proposal,  had  the  contractor’s 
description  for  each  of  the  identified  contracts,  the  administrative 
data,  and  the  offeror’s  own  assessment  of  its  relevant  experience.  A 
second  book  included  all  the  assessment  reports  and  (Questionnaires  for 
the  contracts  that  the  offeror  had  worked  on.  The  questionnaires  and 
assessment  reports  were  organized  in  the  same  manner  as  the  offeror’s 
proposal.  Pages  were  numbered  for  the  second  book  in  such  a  way  so  that 
the  assessment  reports  and  questionnaires  coincided  to  the  contracts 
identified  by  the  offeror.  A  third  book  was  put  together  for  each  PRAG 
member  for  taking  notes  arid  facilitating  the  assessment  process  (7;  15; 

19;  23;  24;  44). 

According  to  Lieutenant  Coombs,  feedback  was  mixed  over  the  two 
approaches.  There  were  those  who  would  have  the  secretariat  decide  which 
information  was  relevant  to  each  item,  and  others  wanted  to  see  the 
full  evaluation  and  how  the  different  elements  interrelate  (15).  In 
Lieutenant  Coombs’  opinion,  the  second  approach  was  better,  because  there 
were  times  when  she  did  not  have  the  experience  to  decide  which 
information  was  relevant  to  each  item.  Colonel  Wayne  Frey  agreed  with 
L  i  eut  enant.  Coombs : 

The  original  method  of  organizing  the  data  by  item  involved  a 
lot  of  useless  cut  and  paste  work.  Although  it  seemed  to  be 
convenient  on  the  first  time  through,  we  later  found  out  that  it 
caused  us  to  do  a  lot  of  page  flipping.  (23) 
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Each  PRAG  member  would  sit  down  w'ith  copies  of  the  first  and  second 


books  and  turn  the  pages  together.  The  members  would  read  the 
contractor’s  input,  read  the  government’s  input,  make  their  assessment 
for  each  item,  and  then  move  on  to  the  next  example  (7;  15;  19;  24;  44). 
Each  PRAG  member  would  read  the  entire  set  of  forms  to  determine  the 
salient  information.  Members  wanted  to  read  each  report  in  its  entirety 
because  so  much  of  the  information  was  overlapping  (7;  24).  Members  did 
not  want  data  averaged  or  in  any  way  processed  for  them,  although  many 
used  their  own  weighting-  techniques  (7;  24).  Evaluations  were  documented 
on  forms  which  included  lines  to  write  comments  and  a  box  to  identify  the 
risk  (7;  15;  19;  24;  44).  Examples  of  the  forms  used  are  shown  in  Figure 
4-1.  Item  Assessment  Sheet  and  Figure  4-2.  Final  Risk  Ratings. 


Offeror:  _ _ 

Program  Title: 


I  tern 


CPAR/ 

Ques  t i onna i re 


Contract  No: 


Member  Rating  and  Remarks 


T.l 

T.  2 

M.  1 

C.  1 


Figure  4-1 .  Item  Assessment  Sheet 
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Offeror : 


ITEM  RISK  REMARKS 

T.l  .  . 


T.2 


M.  1 


C 


Figure  4-2.  Final  Risk  Rat  ings 

[n  some  cases  the  PRAG  members  were  divided  into  teams  to  evaluate 
each  area.  Members  would  sit  on  two  different  area  teams.  Meeting’s  were 
held  for  each  area  team  to  consolidate  the  item  assessments  (7;  19;  24; 

14 ) .  A  consensus  would  result  of  either  high,  moderate,  or  low  risk  for 
each  item.  If  there  was  not  enough  information  to  support,  an  item,  an 
assessment  of  Not  Applicable  would  be  app  1  i  d.  'Idle  item  assessments  were 
then  discussed  and  rolled  up  to  an  area  assessment.  The  discussions 
revolved  around  the?  strengths  or  weaknesses  of  the  contractor  (7;  35;  19; 
21;  14). 
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In  a  less  formal  PRAG,  the  assessment  reports  and  ques t i onna i res 
were  simply  placed  into  folders;  one  for  each  offeror  (18).  Members 
reviewed  each  item  within  their  area  of  expertise  and  presented  their 
assessment  to  the  rest  of  the  PRAG.  Other  members  asked  questions  and 
added  information  resulting'  in  a  group  consensus  (18;  15). 

Ballistic  Systems  Division.  At  Ballistic  Systems  Division 
a  matrix  was  used  to  catalogue  the  information  and  as  a  tool  to  do  the 
assessment  (6;  12;  36).  An  example  is  shown  below. 


TITLE  CONTRACT  PERIOD  OF  VALLE/ 
NUMBER  PERFORM-  TYPE 
ANCE 


RISK 


C  T/M 


SOURCE 


COMMENTS 


Figure  4-3.  Ballistic  Systems  Division  Performance  Data  Matrix 

PRAG  members  worked  either  individually  (36;  37)  or  as  a  group  (12) 
to  evaluate  the  data  from  the  (.PARS  and  from  questionnaires.  Folders 
were  created  for  the  offerors  and  their  subcontractors  to  sort,  the 
ques  t.  i  onna  i  res  ( 37  )  . 
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When  reviewing  the  assessment  reports  and  questionnaires,  each  PRAG 
member  looked  at  the  description,  the  color  ratings,  and  the  narrative. 
The  descriptive  information  was  used  to  separate  the  questionnaires  and 
assessment  reports  according  to  the  screening  criteria  (6).  Each  member 
read  through  the  assessment  reports  and  questionnaires  to  find  relevant 
pieces  of  information  that  pertained  to  the  effort  and  wrote  them  down  in 
bullet  format  on  the  matrix  (6;  12;  36).  It  was  a  subjective  process 
based  on  the  judgement  of  the  PRAG  member.  The  colors  were  not  averaged, 
but.  rather  the  description  and  narrative  were  used  to  get  at  the  heart  of 
vhai  the  person  who  evaluated  the  contractor  was  driving  at  (36). 

Electronic  Sys t, ems  Division.  At  Electronic  Systems 
Division,  a  slightly  different  format  of  a  matrix  was  used  (Figure  4-1). 
Under  the  three  areas  (cost,  technical,  and  management ) ,  comments  from 
assessment  reports  and  questionnaires  were  consolidated  with  succinct 
statements  describing  how  the  offeror  had  performed  (5;  38).  Information 
was  extracted  from  the  assessment  reports  according  to  the  items  which 
were1  in  the  (Questionnaire  (5). 


Program  Dollar  Period  of  Cost  Technical  Management 

Name  Value  Performance 


Figure  4-4.  Electronic  Systems  Division  Performance  Data  Matrix 
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PRAG  members  looked  at  data  subjectively  using  their  best  experience 
and  judgement  (5;  38)  and  highlighted  any  areas  which  showed  outstanding 
or  less  than  satisfactory  performance  (45). 

Muni t ions  Systems  Division  At  Munitions  Systems  Division 
the  data  collected  was  compiled  using  one  of  three  different  methods.  In 
one  case,  the  PRAG  members  sat  down  together  and  completed  a  CPARS  form 
from  the  data  gathered  during  the  interview  (29;  49).  The  PRAG  then  used 
Block  14  from  the  CPARS  form  to  come  up  with  the  overall  rating's  for  the 
evaluat ion  areas  (49) . 

In  another  PRAG,  the  individual  who  performed  the  interview  rated 
the  contractors  performance  (blue,  green,  yellow,  or  red)  for  each 
question  and  then  wrote  a  short  paragraph  summarizing  the  key  points  for 
that  contract  Ml).  The  PRAG  members  wrould  look  at  notes  and  identify 
the  rating  and  any  derogatory  or  other  pertinent  comments  from  the 
interviews  (41). 

In  the  last  case,  assessment  reports  and  quest ionnaires  were  grouped 
according  to  contractor  and  the  PRAG  met  and  discussed  the  differences 
based  on  the  various  sources  of  information  (57).  Members  looked  at 
rating's  given  on  each  assessment  report  and  the  grading  scales  from 
surveys.  PRAG  members  then  rated  the  contractors  on  the  specific  areas 
identified  on  the  CPARS  form  with  a  scale  similar  to  that  used  on  CPARS 
(28;  57'.  An  example  of  the  form  used  to  document  the  ratings  is  shown 
in  Figure  4-5:  Munitions  Systems  Division  Contractor  Rating  Sheet. 

PR\Gs  looked  for  trends  across  ratings  (16;  29;  32).  If  all  ratings 
were  green,  the  offeror  was  given  the  benefit  of  the  doubt;  however,  if  a 
red,  yellow,  or  blue  rating  appeared,  PRAG  members  tried  to  find  good, 
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Figure  4-5.  Munitions  Systems  Division  Contractor  Rating  Sheets 


solid  justification.  A  yellow  or  red  rating  would  have  to  appear  more 
than  once  for  it  to  be  significant  (16;  32).  One  member  found  that  a 
company  may  be  very  sound  in  one  area,  but  weak  on  another,  and  that  the 
strengths  and  weaknesses  would  be  consistent  across  interviews  (29). 
According  to  Lieutenant  Colonel  Donald  Hutcheson: 

The  entire  team  was  involved  in  the  process.  The  first  part  of 
the  process  was  to  understand  what  the  scope  of  the  contract  was 
that  the  [assessment  report]  or  survey  was  pertaining  to.  We  needed 
to  understand  the  contract's  relevance  to  the  effort  being 
considered.  Then  we  needed  to  understand  the  meaning  of  the  rating; 
was  there  one  thing  that  caused  a  rat ing  to  be  a  green  verses  a 
blue?  To  do  this  one  would  have  to  read  very  closely  through  the 
narrative.  (28) 

Space  Systems  Division.  At  Space  Systems  Division  each 
member  looked  at  the  different,  sources  of  data  (CPARS,  "Past  Performance" 
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volume,  interviews)  (33),  and  highlighted  the  strong  and  weak  points  (42; 
52).  According  to  one  individual,  there  were  no  hard  and  fast  guidelines 
to  go  about  the  process.  The  member  stated  that  it  was  a  subjective 
evaluation  of  the  data  (52).  Trends  were  identified  across  programs  and 
the  government  representatives  for  a  program  were  contacted  when 
questions  arose  (42;  52). 

Perform  Risk  Assessment .  The  final  performance  risk  assessment 


occurred  at. 

ei ther 

the 

area 

1  eve 1  ( 5 ;  6 ; 

12;  25;  28; 

32; 

36 ; 

37; 

38; 

41 

15;  49;  51; 

56;  57) 

or 

both 

the  area  and 

item  levels 

(7; 

15; 

19; 

23; 

24 

14).  The  DRAG  representatives  described  the  Performance  Risk  Assessment 
as  a  two-step  process  which  first  checked  to  determine  each  program’s 
relevance  to  the  effort  being  considered  (15;  23;  28;  42;  57)  and  then 
subjectively  summarised  across  all  contracts  so  that  a  risk  assessment 
for  each  area  could  be  assigned  (5;  6;  7;  19;  28;  36;  38;  42;  44). 

One  PRAG  at  Ballistic  Systems  Division  took  a  different  approach  to 
the  performance  risk  assessment.  It  chose  not  to  provide  an  overall 
performance  risk  assessment  across  contracts.  Instead,  screening 
criteria  were  used  to  sort  the  data  in  several  different  ways  and  risk 
assessments  were  shown  at  the  area  level  for  each  contract  (6;  36). 

The  first,  cut  looked  at  full  scale  development  programs  from  all 
sources.  The  relevant  data  was  extracted  and  written  on  a  summary  form, 
similar  to  the  one  shown  below,  (Figure  4-6)  with  supporting  information 
on  a  second  form  (Figure  4-7)  (6;  36). 

For  the  second  cut  the  screening  criteria  were  more  specific.  lhe 
screening’  criteria  were  firm  fixed  price  contracts  which  were  software 
intensive  and  in  full  scale  development.  The  more  specific  criteria 
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Figure  4-6.  Ballistic  Systems  Division  Risk  Assessment  Summary 
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Figure  4-7.  Risk  Assessment  Supporting  Information 

provided  for  a  much  smaller  data  base.  Again  the  data  was  extracted  and 
put  on  the  forms.  This  provided  both  a  broad  and  refined  cut  at  the 
performance  data  (6;  36). 

The  final  cut  was  to  include  only  the  CPARS  information.  PRAG 
representatives  stated  that  they  found  that  CPARS  was  a  better  source  of 
data  to  evaluate  past  performance.  A  corporate-wide  look  at  the 
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contractors  (a  look  beyond  the  particular  profit  center  which  had 
submitted  the  proposal)  was  taken.  For  the  corporate  view,  cost, 
management,  and  subcontractor  management  were  examined  (6;  36). 

Ident i fy  Concerns  to  Procurement  Contract inff  Off icer  for 
Discussions  With  Contractor.  As  part  of  the  discussion  process,  areas  of 
concern  were  submitted  to  the  offeror  by  letter  from  the  Procurement 
Contracting  Officer.  Concern  could  be  based  on  poor/marginal  performance 
or  inability  to  find  any  experience  relative  to  a  given  item.  Lack  of 
relevant  experience  was  listed  as  a  concern  only  to  make  sure  that  the 
PRAG’s  research  had  not  missed  anything.  The  letter  was  processed  just 
like  a  clarification  request  or  deficiency  report,  although  it  was 
topically  processed  after  the  clarification  requests  and  deficiency 
reports  liad  been  sent  (7;  15;  19;  23;  24;  44;  45;  49;  57). 

Concerns  which  were  highlighted  in  assessment  reports  were  not  sent 
to  the  contractor  because  the  contractor  had  already  had  the  opportunity 
to  respond  (44). 

The  offeror  was  given  between  one  (15;  23;  19;  24)  and  two  weeks 
(44)  to  provide  a  response  Contractor’s  response  was  limited  to  ore 
double-spaced  page  on  each  concern  (7;  15;  19;  23;  24;  44). 

PRAG  representatives  stated  it  was  essential  that  once  the 
government  made  its  evaluation,  it  got  these  concerns  back  to  the 
contractor.  When  the  contractor’s  response  was  received,  the  entire  PRAG 
(15;  23;  19;  24),  or  a  smaller  group  such  as  chairman,  assistant  and  team 
leaders  (7;  44)  would  look  at  the  assessments  and  determine  whether 
modifications  were  necessary. 


Brief  Source  Selection  Authority.  The  Source  Selection 
Authority  is  most  often  briefed  by  the  PRAG  chairman  (5;  6;  7;  10;  12; 

13;  15;  19;  25;  32;  33;  34;  38;  41;  42;  44;  49;  52;  57).  At  Aeronautical 
Systems  Division,  a  rigorous  process  for  the  development  of  the  final 
briefing  has  been  established.  Comparable  levels  of  detail  were  not 
captured  through  the  interview's  w'ith  representatives  from  the  other 
product  divisions.  The  process  as  described  by  representatives  from 
Aeronautical  Systems  Division  is  presented  below  (7;  10;  13;  15;  19;  44). 

At  Aeronautical  Systems  Division,  the  PRAG  chairman  and  secretariat 
develop  a  preliminary  set  of  charts.  The  entire  PRAG  meets  to  review  the 
final  assessments.  The  chairman  and  the  secretariat  then  polish  the 
charts  and  script  for  the  presentation  to  the  Source  Selection  Authority. 
The  PRAG  typically  meets  one  more  time  for  a  dry  run  briefing  to  make 
sure  that  all  the  right  words  and  examples  have  been  captured.  The  fina1 
briefing  to  the  Source  Selection  Authority  provides  the  assessment  of  the 
offerors'  past  performance  for  each  area.  Supporting  information  is 
provided  at  the  item  level.  The  same  areas  and  items  used  to  evaluate 
the  offeror’s  proposal  are  used  for  the  performance  risk  assessment. 
Lieutenant  Colonel  Poleski  stated: 

It  is  not  important  that  the  Source  Selection  Aiithority  get.  the 
message  that  the  offeror  is  a  low,  moderate,  or  high  risk  based  on 
its  past  performance.  What  is  important,  is  that  for  specific  items 
or  areas,  an  assessment  of  the  offeror’s  experience  base  shows  the 
offeror  to  be  high,  moderate,  or  low  risk.  The  performance  risk 
assessment,  combined  with  the  proposal  rating  and  risk,  should 
provide  the  Source  Selection  Authority  with  the  informat ion  needed 
to  make  an  integrated  assessment  of  the  offeror’s  capability  to 
perform  the  work  at  hand.  (44) 

Computer  Support .  PRAG  representatives  were  asked  to  describe  what 
computer  system  capabilities  were  currently  being  used  to  support  the 
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performance  risk  assessment  process.  Computers  were  primarily  used  for 

word  processing  (identified  by  25  of  29  interviewed)  and  briefing 

preparation  (identified  by  23  of  29  interviewed).  Other  functions 

identified  were  data  base  management  for  questionnaire  tracking  (12); 

on-line  templates  for  building  reports,  sections  of  the  Request  For 

Proposal,  and  matrices  (37);  and  use  of  spreadsheets  to  create  contractor 

lists  (15).  The  only  software  package  identified  by  more  than  one  PRAG 

member  was  Harvard  Graphics  (named  by  9  of  29). 

Objective  2.  Determine  the  extent  to  which  information  derived  from 
a  fully  operational  Contractor  Performance  Assessment  Reporting'  System 
(CPARS)  can  support  the  performance  risk  assessment  process; 

Investigative  Question  2.  What  portion  of  the  required  information 
can  be  derived  from  the  data  resident  in  CPARS,  and  how  can  the  data 
resident  in  CPARS  be  processed  to  provide  this  information? 

Informat  ion  Requirements  Supported  by  CPARS ■  The  PRAG 

representatives  were  asked  to  list  each  of  the  specific  types  of 

information  which  were  needed  to  perform  a  risk  assessment.  Each  member 

was  then  asked  to  determine  if  the  identified  types  of  information  could 

he  supported  by  the  CPARS. 

The  different  information  types  have  been  listed  in  Tables  4-1 
through  4-5  according  to  the  evaluation  area  to  which  they  pertained. 

Some  of  the  information  types  could  be  traced  to  the  CPARS;  others  could 
not.  Those  types  which  directly  correlated  with  the  areas  in  Block  14  of 
the  (PARS  form  have  been  boldfaced.  Information  types  which  might  be 
traceable  to  the  CPARS  form  have  been  grouped  under  the  boldfaced  items. 
Informat  ion  types  which  could  not  be  traced  directly  to  a  category  on  the 
CPARS  form  have  been  listed  beneath  the  heading  "Other." 
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Tab 1 e  4-1 


Technical  Evaluation  Area  Information  Types 


Engineering  Des ign/Support  (5;  15;  16;  23;  25;  28;  29;  41;  44;  45;  49; 

Ol  ) 

-  Alternative  solutions  to  problems  (52;  56) 

Correction  of  deficiencies  (56) 

Engineering  Change  Proposal  processing  time  (56) 

Initiate  solutions  to  problems  (19;  52) 

Level  of  competence  (37) 

Requirements  Analysis  (5;  19;  36) 

System  Design  (5) 

Sys  t  ems  Eng i neer  i ng  ( 7 ) 

System  Integration  (6;  36) 

Technical  analysis  (37) 

Product  Assurance  (7;  10 ;  13;  15;  16;  19;  23;  28;  29;  41;  44;  45;  49; 
57) 

-  Environmental  Performance  (10;  13) 

Installed  System  Performance  (13) 

-  Quality  Engineering  (10;  13) 

-  Quality  of  product  (37) 

Software  Development  (5;  6;  15;  16;  23;  28;  29;  36;  41;  44;  45;  49;  57) 

-  Computer  Resources  (7) 

Test  and  Evaluation  (5;  16;  19;  25;  28;  29;  37;  41;  45;  49;  56;  57) 
Other 

Course  development  (24;  44) 

Incorporation  of  commercial  off  the  shelf  and  non  development  items 
into  the  overall  design  (6;  36) 

Instruction  System  Development  (24) 

Plan  for  Rel iabi 1 i ty/Ma intainabi 1 i t y/Produc i b i 1 i t y  (5;  44) 

Prototype  Management  (5) 

-  Training  Delivery  (7;  24) 

Training  System  Concept  Definition  (7;  24) 

Lser  involvement  (6;  36) 
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Table  4-2 


Managemen t  Eva  1  ua t  i on  Area  Inf oma t  i on  Types 


Responsiveness  (6;  i 6 ;  24;  28;  29;  36;  41;  42;  44;  45;  49;  52;  56) 

Flexibility  (52) 

Initiative  (37) 

Proactive  approach  (37) 

Promptly  resolves  issues  (19) 

Schedule  (5;  6;  13;  16;  24;  28;  29;  36;  37;  41;  42;  44;  45;  49;  51;  56) 

-  Master  Integrated  Schedule  (7;  19) 

Meeting  Major  Milestones  (19) 

Subcontractor  management  (5;  6;  16;  19;  24;  25;  28;  29;  36;  37;  41;  42; 
45;  49;  57) 

Competent  resources  (5;  52) 

Other 

-  Acquisition  Support  (7;  15;  23) 

Conf  igurat. ion/Data  Management  (10;  13;  19) 
frigineer ing  Change  Proposal  Process  (24) 

-  Focus  on  key  problem  areas  (52) 

-  Organizational  Control  (24) 

-  Organizat ional  Resources  (7) 

Overall  planning  and  management  (37;  42;  56) 

-  Quality  of  Contract  Data  Requirement  List  deliveries  (19;  42) 

Risk  Management  (5;  7;  19) 

Technical  Management  Planning  (52) 

Timely  Contract  Data  Requirement  List  delivery  (19;  42) 

-  Top  Management.  Participation  (37) 


Table  4-3 

Manufacturing  Management  Evaluation  .Are.  Information  Types 


-  Manufacturing  (16;  15;  23) 

Produc ibi 1 i t y  (13;  56) 

Program  Management.  (13;  15;  23) 
Quality  Assurance  Program  (18;  56) 
Second  Sourcing  (13) 

-  Warranty  Management  (18;  13) 

Work  Measurement  (56) 
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Table  4-4 


Logistics  Supportab i 1 i ty  Evaluation  Area  Information  Types 


Integrated  Logistics  Support  (16;  28;  29;  41;  45;  49;  57) 

Facilities  (37) 

Implementation  (24) 

Maintenance  Planning  (5:  44) 

Spares  Availability  (19) 

Support  (5;  19) 

Support  Requirements  (19) 

Planning  (19;  15;  19;  23;  24) 

Technical  Data  (15;  23;  56) 

Other 

Field  Availability  Rates  (indication  if  availability  is  greater  or 
less  than  what  the  contract  calls  for)  (44) 

Life  Cycle  Costs  (19;  13;  56) 

Logistics  Support  Ana lysis  (15;  23;  56) 

Service  Reports  and  Material  In-processing  Review  Boards  (reporting 
mechanism  once  an  item  is  in  the  field*  (56) 

Supportab i 1 ity  (7;  13) 


Table  1-5 

Cost  Evaluation  Area  Information  Types 


Cost  Performance  (6;  19;  16;  28;  29;  36;  41;  42;  45;  49;  56) 

-  Cost  (7;  15;  23;  24;  44) 

Cost  control  (37) 

Cost  management  (19) 

Elements  of  cost  proportional  to  targets  (37) 

History  of  buy-in’s  \37) 

Cost  Variance  (5;  16;  28;  29;  37;  41;  45;  49;  52;  57) 

Other 

Show  cause  or  stop  work  activities  (19) 

Use  a  reasonable  and  rational  budget  (5) 

Engineer ing  change  proposals  well  prepared  (5) 
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As  can  be  seen  in  the  tables,  there  were  many  different  types  of 
informal' ion  which  could  not  be  traced  to  one  of  the  categories  listed  in 
Block  14  of  the  CPARS  form-  This  same  finding  was  expressed  by  several 
individuals  during  the  interviews  (5;  7;  10;  56).  According  to  Mr. 

Edward  Wallace: 

Other  than  the  few  information  types  which  are  substantial  and 
general  enough  to  have  a  close  match  with  a  block  on  the  assessment 
report,  these  information  items  are  the  sort  of  things  that  would 
not  be  in  the  CPARS.  Assessment  reports  are  written  at  a  much 
higher  leve1  than  the  quest,  ionne  ires .  The  only  time  you  might  get 
these  types  of  information  is  if  the  Program  Manager  thought  it  was 
important  enough  to  write  it  up  in  the  narrative  section.  (56) 

To  get  around  the  problem  associated  by  the  mismatch  between  the 

required  information  and  that  which  could  be  supplied  by  CPARS,  PRAG 

members  had  to  adapt  to  the  system  and  take  advantage  of  its  features. 

According  to  the  PRAG  representatives,  the  descriptive  blocks,  color 

ratings,  and  narrative  sections  of  the  assessment  report  must  be  used 

together  to  develop  the  risk  assessment.  (7;  10;  13;  23;  21;  36;  42;  56). 

When  evaluating  an  offeror,  PRAG  members  would  read  the  description  block 

to  determine  relevancy  and  gauge  the  complexity  of  the  program  (23;  56). 

Members  would  then  search  the  ratings  and  narrative  sections  for 

informal  ion  which  was  relevant  to  the  specific  evaluation  items  and  areas 

(7;  10;  13;  23;  24;  56).  The  PRAG  representatives  stated  that,  they  found 

i  neons  i  .->fenc  ies  in  the  assignment  of  color  ratings  from  evaluator  to 

evaluator  which  made  I  he  narrative  even  more  important  (a  blue  rating  to 

one  person  may  tie  a  green  rating  to  someone  else)  (24;  56).  The  PRAG 

representatives  used  colors  to  find  trends  in  the  information  but  relied 

lu 'wily  on  the  narrative  portion  to  substantiate  the  ratings  and  show- 

re  levanoy  (7;  10;  13;  15;  23;  24;  36;  42). 
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Assessment  Reports  and  Quest,  ionnai  res .  Each  PRAG  representative  was 
asked  to  describe  the  extent  which  CPARS  supported  the  performance 
assessment  process  and  identify  any  differences  between  the  information 
supplied  by  assessment,  reports  and  questionnaires/interviews. 

Information  extracted  from  the  CPARS  provided  6tK%  or  less  of  the 
total  information  requirement,  in  14  of  the  16  PRAGs  investigated  (5;  6; 

7;  10;  12;  Id;  15;  16;  23;  23;  28;  32;  36;  37;  38;  41;  44;  45;  49;  51; 

56;  57).  The'  remainder  of  the'  information  was  gathered  through 
quest ionnai res ,  interviews  and  other  sources.  PRAG  members  anticipated 
continued  use  of  questionnaires  and  interviews  for  collecting'  information 
on  contractors  which  are  not  tracked  by  CPARS  and  on  contracts  which  are 
managed  by  other  services  <12;  13;  44).  Also,  questionnaires  are  useful 
fo  re  Mect  information  from  the  contract  administration  offices  (13). 

Several  individuals  called  for  continued  use  of  questionnaires 
because  CPARS  did  not  provide  the  tvpe  or  depth  of  information  required 
to  perform  an  assessment  . 

The  CPAlvS  provides  performance  informat  ion  in  a  general  nature, 
he  needed  to  develop  the  questionnaire  to  get  as  specific  an  input, 
as  possible.  The  [assessment  reports)  did  not.  lend  themselves  to 
air,  specific  area.  An  assessment  report,  may  give  a  general  look  at 
management  an:!  performance,  but  would  not  get  down  to  the  detail 
,  u  ‘dec! .  (10 ) 

Quest  ionnai res  will  remain  useful  even  if  tnere  were  very  many 
assessment  r  -ports  in  the  data  base,  the  reason  being  to  verify 
•  hi  tigs  on  *  he  assessment  report  which  may  not  be  explicit  !y  spelled 
■mi  .  ( 42  ) 


C  'tnmi'tit  s  on  assessment  report  s  were  not  at  a  level  low  enough 
make  the  assessment  .  hi-  would  hav  e  been  hard  pressed  if  (PARS 
was  the  only  source  of  information.  the  questionnaires  ’  ent  into  a 
tot  mor>  detail  than  the  report s  from  the  CPARS.  (12) 

Sever  a 4  of  the  PRAG  members  interviewed  indicated  t  hat  there  was 


v  mie  ff'erenee  in  tin  frankness  of  comments  between  an  assessment  report 


and  a  questionnaire  for  the  same  contract.  For  many  PR\Gs,  interviews  or 


questionnaires  were  not  sent  when  an  assessment  report  existed.  In  these 
casts  a  general  comparison  across  contracts  was  made  by  the  PRAG 
represent at ive. 

PRAG  representatives  observed  differences  in  comments  made  during' 
interviews  or  on  quest ionnai res  ana  those  made  in  the  narrative  sections 
of  the  assessment  reports.  Several  individuals  felt  that  the  process 
which  the  assessment  report  must  go  through  causes  a  filtering  of  the 
comments  (42).  Some  PRAG  members  stated  that  comments  on  the  assessment 
reports  appeared  "watered  down"  (10;  16;  24;  42),  while  information 
collected  directly  from  the  people  involved  was  very  candid.  One  member 
stated  that  through  interviews  and  questionnaires,  the  government  points 
of  contact  stated  exactly  what  the  problems  were  and  who  was  responsible 
(42).  Another  representative  felt  the  assessment  reports  were  "polished" 
or  "inflated"  when  compared  to  questionnaires  and  that  they  didn’t 
provide  all  the  "between  the  1  ires"  type  of  comments  (7). 

In  contrast.,  other  PRAG  members  felt  that  the  CPAR.S,  because  of  the 
process  involved,  was  the  most  credible  of  all  the  sources  (23;  37;  49). 
Two  representatives  stated  that,  in  their  opinion,  sometimes  a  government 
point,  of  contact  would  liave  an  "axe  to  grind"  and  the  emotionalism  could 
be  seen  in  the  words  (23;  49).  Comments  from  interviews  and 
quest  ionnai res  would  have  to  be  "taken  with  a  grain  of  salt"  (23). 

Another  individual  stated  that  the  fact  that  the  contractor  had  an  option 
to  rebut  or  support  the  program  irvuiager ’ s  comments  increased  the 
assessment  report’s  credibility  (37). 
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Many  members  felt  that  it  was  the  depth  of  information  which  could 
be  extracted  from  the  source  rather  than  frankness  of  comments  that  made 
CPARS  more  er  less  valuable.  PRAG  members  were  divided  as  to  which 
source  provided  more  information.  Several  individuals  felt  that  a 
greater  understanding  could  be  achieved  in  an  intervir"  'r  questionnaire 
(12;  28;  52;  57),  while  others  felt  the  assessment  report  provided  moiv- 
information  (6;  19;  24;  36). 

System  C'apab  i  I  i  t  ies  .  Each  individual  was  asked  what  capabilities  an 
automated  version  of  CPARS  should  have,  and  also  to  identify  any  items 
that  a  designer  of  such  a  system  should  either  concentrate  on  or  avoid. 

The  system  capabilities  as  described  by  the  FRAG  representatives 
take  several  different  forms.  Requirements  fall  into  six  of  the  seven 
activities  inherent  in  the  performance  assessment  process  as  well  as 
general  considerations  for  the  operation  of  tb  ■  system.  In  order  to 
maintain  clarity  and  facilitate  insertion  irto  the  Requ i rement s  Baseline, 
capabilities  have  been  grouped  into  one  of  the  specific  activities  for 
■'■hi eh  they  apply:  Data  Collection,  Data  Input,  Maintenance,  Analysis, 
(Xitput,  and  Protection  of  Information,  or  the  broad  category  of 
Considerations  and  Goals  of  the  System. 

Data  Co  1 1 ec t ion.  There  are  two  major  points  which  must  be 
considered  when  looking  at  the  capabilities  required  to  support  the  data 
collection  function  of  the  system.  They  are  ease  of  input,  ana  adequate 
dept  h  1  i  i  nt'ormat  i  on . 

Lieutenant  Colonel  tko-r ingr on  described  ease  of  input  as  the  need  to 
"make  sure  that  there  are  easy  ways  to  get  the  irrormat ion  into  t ne 
system”  (25).  He  further  stated  that  "it  is  going  to  he  a  big  enough 
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problem  just  to  get  the  program  managers  to  fill  out  the  assessment 
report"  (25). 

The  key  to  providing'  an  adequate  depth  of  information  is  emphasis  on 
the  importance  of  the  Program  Di rector /Program  Manager  narrative  (7;  24; 
44).  The  PRAG  must,  know  the  basis  for  color  ratings  and  their  relevance 
in  order  to  make  an  assessment  (6;  16;  19;  36;  38;  42;  57).  Ensuring 
that  the  system  captures  the  narrative  is  only  part  of  the  solution.  The 
other  part  requires  some  method  to  motivate  Program  Directors /Program 
Managers  to  provide  a  greater  level  of  information  in  the  narrative 
h 1 ock .  l-o  sugges t i ons  inc ! ude 

use  of  standard  type  questions  that  could  be  answered  to 
provide  a  greater  amount  of  detail  in  the  remarks  (41),  and 

an  on-line  description  of  the  CPARS  evaluation  areas.  This 
would  allow  a  comparison  of  the  item  descriptions  to  the  individual 
blocks  on  the  CPARS  form.  (15) 

Another  problem  associated  with  depth  of  information  is  the  ability 
to  determine  the  nature  and  phcise  of  the  program  (6;  28;  37).  This 
information  is  not  always  diseernable  from  blocks  12  and  13  of  the  CPARS 
form  (23).  A  suggested  solution  to  this  problem  is  to  use  key  words 
which  would  describe  the  type  of  contract  (28),  the  phase  of  the  process 
(37),  and  the  type  of  work  performed  (6). 

Data  Input  .  Two  points  were  ......  pertaining  to  the  input  of 

the  data  into  the  system.  The  first  concerned  the  need  to  make  sure  the 
reports  in  the  database  were  the  most  current  (25;  28).  «>ie 
represent  at ive  noted  that  the  members  knew  of  assessment  reports  which 
'-ere  in  the  system,  but  wore  not  in  the  files  yet  (28).  The  second  point 
concerned  the  need  to  update  the  system  and  how  the  database  at  each 
product  division  could  be  updated  (32). 
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Maintenance .  There  were  two  basic  maintenance  functions  which 
were  identified  by  the  representatives  interviewed.  The  first  deals  with 
the  problem  created  by  firms  changing-  names  because  of  mergers, 
takeovers,  and  buy-outs  (15;  23).  One  individual  stated  that  such 
changes  make  it  very  hard  to  know  whom  you  are  talking  about  anymore 
(23).  The  system  needs  to  provide  a  way  to  keep  track  of  name  changes  so 
that  the  individual  PRAG  member  does  not  have  to  (15;  23) 

The  second  maintenance  capability  looked  at  some  sort  of  automatic 
purge  of  old  assessment  reports.  It  was  suggested  that  the  information 
would  not  be  useful  after  five  years,  and  the  value  of  a  historical 
record  wou  1  d  tie  ques  t  i onab  1  e  ( 56 ) . 

Analys is .  Two  broad  analysis  capabilities  need  to  be  provided 
by  the  system.  These  are  the  ability  to  determine  which  assessment 
reports  are  relevant,  and  the  ability  to  provide  a  description  of  the 
data  available  on  each  contractor. 

Relevancy.  Tn  order  to  determine  if  an  assessment  report 
is  re  levant  to  the  program  under  consideration,  the  PRAC  rcprcscr.fat i\ os 
suggested  the  capability  be  provided  to  search  through  the  CPARS  by 
various  criteria.  The  requested  search  capabilities  would  locate 
information  which  is  identified  in  standard  fields  and  on  information 
which  may  or  may  not  be  provided  within  certain  fields  on  the  CPARS  form, 
key  fields  already  identified  by  the  CPARS  form  include: 

Parent  Corporation  (23;  37;  15) 

Contractor  Division  (23;  37;  44;  45) 

Type  of  Report  (25) 

Contract  Number  (15) 
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Product  Di vis. ’on  (42;  45) 

Period  of  Performance  (16;  15) 

Range  of  Dollar  Value  (6;  37;  42;  52  * 

-  Contract  Type  (6;  12;  23;  32;  37;  42;  49) 

Information  that  may  be  provided  within  a  field  on  the  CPARS  form 

but  which  does  not  have  a  preset  standard  format  includes: 

Nature  of  Program:  software  verses  hardware  intensive  effort, 
integration  type  contract  (12;  28;  32;  44;  52;  57) 

Phase  of  Acquisition  Process:  Production,  Full  Scale 
Development,  Demons t rat  ion /Val i da t ion,  Concept  Definition  (6;  16; 
28;  32;  37;  38;  42;  44;  45;  49  52;  57) 

Program  Description:  Aircraft,  M'ssile,  Radar,  Command  Control 
and  Communications,  Space,  Training  (6;  12;  28;  32;  33;  37;  38; 
44;  15;  52;  57) 

Unique  Aspects  Within  the  Narrative:  Quality  Assurance,  Use  of 
Special  Software  Languages,  Training  (19;  16;  51) 

Data  Base  Descript  ion.  The  other  broad  analysis 

capability  requested  by  the  PRAG  representatives  was  the  ability  to 

provide  a  description  of  the  data  base  for  each  contractor.  Suggested 

descriptive  capabilities  are  listed  below. 

For  each  parent  corporation  and  division/profit  center: 

Provide  frequency  of  reports  by  contract  type  (44;  4^;  49) 

Provide  frequency  of  reports  by  phase  of  acquisition  (45;  49) 

Provide  frequency  of  reports  by  program  description  (45) 

For  each  parent  corporation,  division/profit  center,  and 
contract : 

Provide  an  average  overall  rating  across  assessment  reports 
for  each  evaluation  area  (42) 

Provide  a  f re<juency  of  ratings  across  assessment  reports  for 
each  evaluation  area  (42) 
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Provide  a  frequency  of  assessment  reports  within  preset  cost 
and  schedule  variance  ranges  (44) 


-  Capability  to  describe  trends  over  initial,  intermediate,  and 
final  assessment  reports  for  one  contract  (25;  28;  31) 

Capability  to  describe  trends  across  many  contacts  (28;  42;  45; 
49;  51) 

Output .  The  following  types  of  output  capabilities  were 

requested  by  the  PRAG  representatives.  Output  should  be  available  both 

for  display  on  the  computer  screen  and  in  hard  copy. 

Provide  entire  copy  of  assessment  reports  which  have  been 
determined  to  be  relevant  (7;  13;  15;  19;  24;  28;  37;  42;  44;  56) 

Provide  copy  of  summary  reports  based  on  relevant  fields  of  the 
assessment  report  (19;  32;  37) 

Provide  copy  of  descriptive  statistics  defined  in  analysis 
sect  ion  (19;  32 ;  49 ) 

Download  information  to  the  Performance  Data  Matrix  and  briefing 
charts  (12) 

Protect  ion  of  Informat  ion.  One  of  the  key  features  in  the 
conception  of  the  CPARS  was  its  confidentiality.  Any  automated  system 
developed  to  support  CPARS  and  the  PRAG  needs  to  maintain  this  (7).  An 
automated  data  base,  must  consider  the  protection  of  data  since  it  is 
source  selection  sensitive  (7;  23;  49).  It  should  have  appropriate 
control  features  (7;  49)  to  make  sure  that  the  data  cannot  be  changed 
(56)  or  retrieved  by  someone  who  should  not  have  access.  According  to 
Mr.  Mutrie: 

Security  of  the  system  is  absolutely  essential.  There  has  to 
be  some  kind  of  control  system  to  allow  free  access  to  people  who 
r«ed  t l'e  information,  but  not  allow  them  to  change  any  of  the 
information.  (38) 
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Cons ide rat  ions  and  Goals  of  the  System.  An  important 
consideration  for  the  development  effort  raised  by  Lieutenant  Colonel 
Poleski  is  that  the  mechanisms  purposely  put  into  the  CPARS  to  ensure  the 
integrity  of  the  information  must  not  be  sacrificed.  These  include  the 
four  step  assessment  report  development  process  of  Preliminary 
Assessment,  Contractor  Review,  Program  Manager  Update,  and  examination  by 
the  Reviewing  Official.  Lieutenant  Colonel  Poleski  states: 

Hie  Department  Of  Defense  Contract  Performance  Evaluation 
system  did  not  have  the  checks  and  balances  which  the  current  system 
now  has,  and  thus  the  information  was  less  reliable.  We  can’t  lose 
any  of  these  checks  and  balances.  (44) 

Lieutenant  Colonel  Poleski  was  referring  to  one  of  the  problems 
discussed  earlier  which  plagued  the  now  defunct  Contractor  Performance 
Evaluation  program.  That  is,  there  were  problems  associated  with 
ensuring  objectivity  of  the  program  manager  evaluations  (35:15).  It  is 
vital  that  any  automation  attempt  help  assure  the  information  objectivity 
and  integrity. 

These  sentiments  are  reinforced  by  comments  made  by  other  PRAG 
members  (2;  38;  52).  Mr.  Mutrie  of  the  MITRE  corporation  stated: 

The  contractor’s  comments  are  very  important  because  it 
provides  a  closed  loop.  The  PRAG  member  can  feel  confident  with  the 
information  on  the  form.  [We]  need  to  make  sure  this  still  works 
when  automating  [’he  system].  (38) 

An  important  consideration  raised  by  Colonel  Prey  concerned  the 
;unount  of  work  necessary  to  transit  r  the  existing  data  into  the  automated 
data  hast'  (23).  He  stated  that  the  just i f i cat  ion  for  development  and 
institution  of  the  new  system  must  keep  this  significant  effort  in  mind 
(23).  Since  the  life  of  an  assessment  report  is  relatively  short  (five 
years),  a  phased  implementat ion  similar  to  what  is  currently  being 
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accomplished  with  the  CPARS  and  use  of  questionnaires  could  effectively 
diminish  any  potential  problem. 

Several  requirements  for  an  automated  system  were  identified  by  the 
PRAG  members  which  cut  across  all  boundaries  of  the  system  and  describe 
the  attributes  a  workable  system  should  have.  For  this  reason,  the 
requirements  have  been  listed  below'  as  general  goals  of  the  system. 

The  system  must  be  user  friendly  (19;  23;  25)  and  "Murphy 
Proofed"  (25).  It  should  be  menu  driven  (32),  and  provide  the  right 
tools  and  amount  of  hardware  to  do  the  job  (19;  45).  It  cat  Loot  make 
the  task  of  the  PRAG  tiny  more  tedious  than  it  alreac.,.  is  (23). 

The  system  must  provide  as  easy  access  to  the  data  as  the  PRAG 
members  have  now  using  paper  copies  or  the  system  of  notebooks  (19; 
25;  45).  Any  automated  system  should  ensure  that  it  does  not  nold 
up  the  PRAG  member  (19;  45). 

The  system  needs  to  be  flexible  enough  so  that  it  can  be 
expanded  as  the  system  matures  (56). 

The  system  should  limit  the  amount  of  paper  necessary  for  the 
PRAG  to  get  its  job  done  (23).  Effort  should  be  put  into  the 
printed  product  and  the  system  should  provide  the  capability  to  sec 
the  finished  product  on  the  screen  (25). 

Other  comments  made  which  do  not  actually  fall  into  the  requirements 
domain,  hut  are  relevant  to  the  development  effort  and  the  final 
acceptance  of  the  system  have  been  included  below.  According  to  Mr. 
Raymond  Albert  from  Electronic  Systems  Division,  the  (PARS  should  not  be 
automated.  He  was  not  very  enthusiastic  about  putting  copies  of 
assessment  reports  into  a  computer  system,  stating  that  the  process  is 
too  subjective. 

Everything  is  avai iable  at  a  glance,  and  l  would  not  try  to 
automate  that.  A  data  base  is  useful  only  if  the  information  is 
requested  often,  or  if  it  has  to  be  sorted,  reshuffled  back  and 
forth,  or  analyzed  somehow.  But  none  of  that’s  true  with  these 
assessment  reports.  I  looked  at  that  a  long  time  ago  when  someone 
wanted  to  automate  [Electronic  Systems  Division’s  Contract 
Performance  Evaluation  Program]  data  base.  All  that  you’re  going  to 
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do  is  spend  a  whole  lot  of  time  typing1  them  all  in  and  they’re  going' 
to  sit  there  for  months  and  some  even  for  years,  and  nobody  is  going 
to  look  at.  them.  So  just  take  originals  and  put  them  into  a  file. 
Then  when  you  want  to  look  at  them,  you  look  at  them.  (5) 

When  asked  whether  he  felt  that  an  automated  system  could  have  any 

merit  at  all,  Mr.  Albert  replied: 

I  would  not  categorically  say  that  it.  is  a  bad  idea;  some 
people  may  have  good  reasons  that  I  don’t  know  about.  But  I  don’t 
have  any  reasons  right  now  why  I  would  want  to  do  it.  (5) 

A  slightly  more  positive  but  still  reserved  view  of  an  automated 

system’s  capacities  was  presented  by  Lieutenant  Colonel  Poleski: 

The  contractor  input  and  any  quest ionna ires  necessary  to 
augment  the  existing  assessment  reports  will  be  on  paper.  Both  of 
these  could  somehow  be  put  into  the  system,  but  I’m  not  so  sure  that 
it  would  be  cost  or  time  effective  just  to  have  the  information  on 
the  computer.  If  the  assessment,  reports  were  on  the  computer,  then 
you  would  have  to  read  a  little  bit  here  and  there.  I’m  coming  to 
the  conclusion  that  the  information  would  have  to  be  all  on  the 
system  or  all  off.  Some  of  the  summarizing  of  the  CPARS  information 
could  be  done  on  a  computer  system,  but  I’m  not  sure  it  would  be  all 
that  much  of  a  savings.  If  there  was  a  way  to  sort  through  the 
information  in  the  data  base,  then  it  would  be  convenient  for  the 
secretariat  [PRAG  assistant).  (44) 

User  of  System.  Given  that  an  automated  CPARS  could  be  developed, 
the  PRAG  representatives  were  asked  who  the  actual  user  of  the  system 
should  be. 

The  response  to  this  question  varied  greatly.  Potential  users  of 
the  system  identified  were:  an  administrative  assistant  (6;  12;  36;  49; 
36),  the  PRAG  assistant,  (secretariat)  (6;  7;  10;  12;  13;  15;  24;  25;  51; 
56),  both  the  PRAG  assistant  and  the  PRAG  chairman  (25),  the  PRAG 
chairman  (16),  any  PRAG  member  (23;  25;  28;  36;  37;  38;  41;  42;  45),  and 
a  representative  from  the  (PAR  focal  point’s  office  (33;  52).  One  member 
stated  that  as  long  as  the  system  is  "user  friendly,"  the  ultimate  user 
should  not  matter  (32). 
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Analysis  of  Interviews  w i th  CPARS  Foca 1  Points 

Interviews  with  the  CPARS  focal  points  were  used  to  describe  the 
implementation  of  the  performance  assessment  reporting  process.  Results 
from  these  interviews  fulfill  the  third  objective  of  this  research 
project.  A  summary  of  the  results  for  each  question  is  presented  below. 
The  actual  questions  have  been  included  in  Appendix  C:  Survey 
Instruments . 

Objective  3.  Determine  the  administrative  procedures  used  to 
collect,  process  distribute,  and  protect  cont ract or  performance 
information  tuider  the  CPARS; 

Investigative  Question  3.  What  are  the  administrative  procedures 
for  collecting’,  processing,  and  distributing  contractor  performance 
informat  ion. 

Administ.rat  ion  of  the  CPARS .  Each  focal  point  was  asked  to  describe 
how  administration  of  the  CP.ARS  uas  organized  at  their  product  division. 

At  Aeronautical,  Munitions  and  Space  Systems  Divisions,  there  is  one 
central  person  responsible  for  the  CPARS.  These  divisions  also  hav< 
established  a  point  of  contact  in  each  of  the  two- let  ter  offices  to  keep 
track  of  contracts  requiring  assessment  reports.  The  two-letter  points 
of  contact  make  sure  the  program  manager  initiates  an  assessment  report 
and  that  all  contracts  which  need  to  be  reported  on  are  (9;  17;  55).  At 
Munitions  Systems  Division  each  point  of  contact  provides  a  quarterly 
report  to  the  CPARS  focal  point  on  the  status  of  reports  (55). 

At  Ballistic  Systems  Division  there  is  a  CPARS  focal  point  mid  an 
administrative  assistant.  There  are  no  other  points  of  contact  (48,  58). 
Electronic  Systems  Division  has  a  CPARS  focal  point,  an  administrative 
assistant.,  cuid  points  of  contact  in  each  two-letter  office  (11;  38). 
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Contractor  Performance  Assessment  Process.  The  CPARS  focal  points 
were  asked  to  describe  the  contractor  performance  assessment  process  as 
it  is  currently  implemented.  The  focal  points  were  asked  to  provide 
the  detailed  steps  and  the  peculiarities  of  the  process  vhich  would  not 
be  found  in  the  CPARS  regulation.  To  facilitate  the  discussion,  the 
process  was  categ'orized  by  five  different  activities:  data  collection, 
data  input,  maintenance,  output,  and,  reporting. 

Data  Collection.  Data  collection  is  the  most  complex  of  the 
performance  assessment  process  activities.  Five  steps  (initiation, 
evaluation,  contractor  comments,  update,  and  review)  were  provided  to  the 
focal  points  as  a  guide  to  help  them  explain  how  the  actual  assessment 
report  is  developed.  Each  focal  point  was  asked  to  describe  any  local 
policies  or  practices  which  were  distinctive  to  their  product  division. 

Process  In i t i at  ion .  The  program  manager  is  not  i f i ed  to 
initiate  tin  assessment  report  on  a  contract  either  directly  l>v  the  CPARS 
focal  point  (  If);  50)  or  throw”’ i  the  two-letter  points  of  contact  (9;  17; 
2(1;  55). 

At  Electronic  and  Munitions  Systems  Divisions  the  CPARS  focal  point 
keeps  a  suspense  file  so  that,  the  program  manager  car.  be  given  75  days 
not  ice  before  the  assessment  report  is  due.  'Hus  provides  30  days  for 
preparation,  3t)  days  for  contractor  to  respond,  and  15  days  for  review 
and  approval  (9;  11;  30). 

At  Munitions  Systems  Division  the  focal  point  gt  ts  a  quarterly 
report  from  the  Acquisit  ion  Management  Informal  ion  System  of  till 
contracts  written  which  are  over  five  million  dollars.  The  two-letter 
points  of  contact  determine  which  of  those  programs  require  assessment 


reports  and  inform  the  program  manager  when  one  is  due.  The  poin’ s  of 
coat  an  t,  give  the  program  manager  the  (PARS  focal  point’s  name  and  phono 
member.  The  CPARS  focal  point  then  gives  the  program  manager  a  copy  of 
the  regulation  and  provides  any  necessary  assistance  (9). 

Once  an  assessment  report  is  initiated  the  focal  point  keeps  a  list 
of  the  day  that  it  is  due  out  to  the  contractor,  the  day  it  is  due  back 
to  the  government,  and  the  day  it,  should  be  signed  by  the  reviewing 
official .  The  focal  point  keeps  track  of  the  assessment  reports  through 
the'  two-letter  points  of  contact.  (9). 

At  Aeronaut  tea  I  Systems  Division,  the  focal  point  sends  a  reminder 
to  the  two- let  ter  points  of  contact  120  days  before  the  assessment 
reports  are  due.  The  reminder  states  which  assessment  reports  *ere 
submitted  last  year  and  provides  a  schedule  of  milestones  which  should  he 
met.  The  two- let  ter  focal  point  then  informs  the  program  manager  to 
initiate  an  assessment  report  (17).  The  assessment  reports  art'  written 
only  three  t  imes  a  year  iri  .April,  August,  and  November  (17). 

At  Space  Systems  Division  updates  are  due  when  the  period  of 
performance  covered  by  the  report  hcis  expired  (26;  55). 

At  Ballistic  Systems  Division,  the  review  cycle  begins  in  Aprii  of 
each  year.  Using  the  Acquis i t  ion  Management  Informat  ion  System  and  the 
established  manual  (PARS  filing  system,  the  focal  po>nt  makes  a 
dr*  erniinat  ion  of  which  contracts  are  c<uididaios  to  be  included  in  CPARS. 

A  clerk  at  Ballistic  Systems  Division  has  developed  a  template  that 
works  with  WordPerfect  to  complete  the  front  page  of  the  (PARS  fo-m.  Duo 
to  problems  with  t he  quality  of  the  reports  during'  the  first  cycle,  the 
focal  (joint,  intends  on  retyping  each  assessment  report.  According  to  Mr. 
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David  Smith  and  Ms.  Pat  Olson,  having'  a  copy  of  the  report  in  an 
automated  form  will  shorten  the  processing  time  during  the  next  cycle. 

The  program  managers  are  prohibited  from  keeping  a  copy  of  the  assessment 
report,  and  having  the  template  and  the  prior  year’s  assessment  report 
will  help  them.  Also,  the'  automated  form  will  maintain  consistency  in 
the  program  descriptions  during  the  life  of  the  contract  ( 50 ) . 

At  Ballistic  Systems  Division,  the  Air  Force  System  Command  Form  125 
is  sent  for  each  contract  from  the  focal  point  to  the  program  director 
(40;  50 1  .  Hie  front  page  of  the  assessment,  report  has  already  been 
completed  through  the  idenlificat  ion  of  the  past  color  in  block  14  by  a 
member  of  the  focal  point’s  office  using  the  WordPerfect  template.  The 
evaluator  makes  appropriate  changes  to  the  existing  information  and 
completes  the  assessment.  Someone  from  the  focal  point’s  office  will 
t  hen  update  the  information  in  the  word  processing'  template  so  that  it 
can  he  used  again  during  the  next  cycle. 

Program  Manager’s  Evaluation.  At  each  product  division 
the  program  manager  has  access  to  Air  Force  Systems  Command  Regular  ion 
P..../-54,  and  the  product  divisions  local  supplement  to  assist  the 
evaluation  process  (ft;  11;  17;  26;  40;  40;  50;  55).  Aeronautical  Systems 
Division  is  also  establishing  a  training’  progT.'urt  for  program  managers 
(17,i.  At  Ballistic  Systems  Division  specific  instructions  are  given  in 
file  letter  to  t  he  program  directors. 

The  program  m;inager  narrat  ive  is  limited  to  the  space  provided  in 
Block  10  plus  one  add i t  ional  page  (17).  The  person  performing  t he 
in  i  t  ia!  assessment  a; id  the  person  signing  the  form  as  the  program 
d  i  ret  t  or  /manager  may  be  different.  The  level  of  this  initial  signature 
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also  differs  across  the  product  divisions,  as  does  the  coordination 
necessary  prior  to  the  report  being'  sent  to  the  contractor. 

At  Space  Systems  Division  the  person  actually  responsible  for  me 
contract  performs  the  evaluation  and  signs  the  form  in  Block  17.  The 
program  manager  then  sends  the  form  under  a  cover  let  ter  to  the 
contractor.  The  CPARS  focal  point  does  not  see  the  report  prior  to  its 
being  sent  to  the  contractor  (26;  55). 

At  Aeronautical  Systems  Division,  the  program  manager  or  project 
officer  actually  in  charge  of  the  contract  will  fill  out  the  assessment 
report  .  The  person  who  signs  Block  17  as  the  program  director/manager 
differs  between  programs.  It  could  be  the  actual  program  manager,  the 
three- let  ter ,  or  two- let  ter.  It  is  very  flexible.  What  has  become 
Inflexible  is  that  the  reviewing  official  must  be  a  general  officer  (17). 

For  Ballistic  Systems  Division  the  lead  project  officer  for  each 
contract  provides  the  initial  evaluation,  the  form  goes  through  the 
project  officer’s  boss  (the  Director  of  Engineering)  for  review,  and  then 
is  reviewed  by  the  program  director  before  being  submitted  to  the 
contractor.  The  office  from  which  the  assessment  report  is  sent  to  the 
contractor  depends  on  the  program  office.  For  some  the  program  director 
will  sign  a  transmittal  letter  which  has  been  prepared  by  the  focal 
point’s  office.  Other  program  offices  will  prepare  their  own  t r?uismi 1 1 a  1 
letter.  In  all  cases,  the  focal  point  or  the  administration  assistant 
will  review  the  assessment  report  prior  to  its  being  sent  to  the 
contractor  (40;  50). 

At  Munitions  Systems  Division,  the  program  manager  evaluates  the 
contractor  and  then  submits  the  initial  assessment  report  to  the 


4-39 


two- let  ter  office  for  review  prior  to  its  being  sent  to  the  contractor 
for  comment.  The  evaluator  is  the  program  manager  in  charge  of  the 
specific  contract  while  the  actual  individual  who  signs  the  form  as 
program  di rector/manager  is  the  two-letter  chief  (9). 

Contractor  Comments.  The  assessment  report  is  sent  to  the 
contractor  by  certified  mail  to  ensure  time  and  date  of  receipt.  At  that 
point  the  contractor’s  30  day  clock  starts.  Contractors  are  asked  to 
provide  at  least  a  signature  and  date  on  the  form.  Comments  are  limited 
to  block  18  plus  one  additional  page  { 9 ;  11;  17;  26;  30;  551. 

The  program  manager  (26;  55),  program  director  (17;  10;  50), 

two-letter  chief  (9)  or  the  CPARS  focal  point  (40;  50)  may  sign  the 
tran«mitta!  ' et  *  cr  to  the  contractor.  At  each  of  the  product  divisions, 
stunple  letters  liave  been  developed. 

Program  Manager  Update.  The  program  manager  reviews  the 
assessment  report  and  determines  if  revisions  are  necessary.  If  the 
program  manager  decides  to  update  the  initial  assessment,  it  is  done  on  a 
new  form  which  is  attached  to  the  original.  If  a  revision  is  necessary, 
the  top  portion  of  the  new  form  is  completed  (items  1  through  5)  and  the 
following  statement  is  entered  in  Block  12  "Revision  to  CPAR  for  period 
(insert  period  covered)."  Blocks  requiring  revision  are  then  updated 
(11;  17;  26;  30;  40;  50;  55). 

Tlie  responsible  party  then  signs  the  form  in  Block  17  .mid  transmits 
it  o  the  reviewing  official.  As  stated  earlier,  this  initial  signature 
may  he  that  of  the  program  manager,  a  three- let  ter  supervisor,  the 
program  director  or  two- let  ter  officer. 
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Rev i ew  Process .  There  is  only  one  reviewing  official  at 


Ballistic,  Electronic,  and  Munitions  Systems  Divisions.  The  reviewing 
official  for  Electronic  Systems  Division  is  the  vice  commander.  From  the 
program  manager,  the  assessment  report  goes  to  director  of  contracting 
for  coordination  prior  to  going  to  the  reviewing  official  for  signature 
(11;  30).  The  reviewing  official  for  Ballistic  Systems  Division  is  the 
commander  (40;  50).  The  reviewing  official  for  Munitions  System  Division 
is  the  vice  commander  (9). 

At  Space  Systems  Division,  the  two- letter  director  signs  the  form  as 
the  reviewing  official.  The  reviewing  official  has  the  opportunity  to 
make  comments  in  Block  20  (26;  55). 

According  to  the  focal  point  at  Aeronautical  Systems  Division,  the 
signature  in  Block  21  of  the  assessment  report  must  be  a  general  officer. 
Tire  reviewing  official  may  be  the  two-letter  officer,  vice  commander,  or 
commander  depending  on  the  program.  The  reviewing  official  makes 
comments  and  then  signs  the  form  (17). 

Data  Input .  Each  focal  point  was  asked  to  describe  how  and  how 
often  each  report  was  entered  into  the  command  wide  data  base.  The  focal 
points  were  also  asked  whether  there  were  any  attachment-?  in  addition  ♦ 
the  continuation  pages  for  the  program  di rector/manager  narrative  or  the 
contractor’s  c  ommen  t s . 

After  the  reviewing  official  signs  the  form,  four  copies  are  made 
and  one  copy  is  sent  to  each  of  the  product  divisions  (11;  17;  26;  30 ; 

JO;  50;  55).  In  most  cases  the  assessment  reports  are  mailed  to  the 
other  product  divisions  as  they  are  completed;  however,  the  focal  point 
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at  Aeronautical  Systems  Division  waits  until  10-12  are  completed,  and 
then  sends  them  out  as  a  batch  (17).  Each  assessment  report  is  double 
wrapped.  The  inner  envelop  is  marked  "Source  Selection  Sensitive  -  For 
Official  Use  Only".  The  outer  envelope  is  marked  "To  Be  evened  By 
Addressee  Only"  (9;  11;  17;  30;  40;  50). 

At  Munitions  Systems  Division  the  focal  point  makes  five  copies  of 
the  assessment  reports  as  they  come  in  and  sends  a  copy  to  each  product 
division  plus  one  to  Air  Force  Systems  Command  (9).  The  other  product 
divisions  send  a  quarterly  or  more  frequent  report  to  Air  Force  System 
Command  describing'  the  latest  assessment  reports  rather  than  sending  the 
actual  assessment  reports  (11;  17;  30;  40;  50).  These  reports  contain 
the  contractor  name,  subsidiary  of  division,  contract  number,  period  or 
performance,  and  a  brief  description  of  the  contract  (11;  17;  30). 

Each  focal  point  must  maintain  a  local  data  base  of  the  assessment 
reports  originated  by  their  own  product  division  plus  copies  of 
assessment  reports  initiated  by  the  other  product  divisions.  The  method 
of  filing  assessment  reports  differed  between  product  divisions. 

At  Ballistic  and  Electronic  Systems  Divisions,  assessment  reports 
are  filed  by  parent  corporation  and  sequentially  by  contract  number.  The 
reports  are  separated  within  each  folder  according  to  the  originating 
product  division  (11;  30;  40;  50).  A  problem  was  raised  '  'he  Ballistic 
Systems  Division  focal  point  concerning  what  name  to  use  t.  »e  the 
assessment  reports.  PRAG  members  may  know  a  contractor  by  one  name  even 
though  the  contractor  is  referred  to  in  the  CPARS  regulation  as  something 
else.  The  potential  problem  is  that  someone  may  come  looking  for  a 
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report  and  not  find  it.  The  focal  point  and  the  administrative  assistant 
have  chosen  to  go  with  the  names  referred  to  in  the  regulation  and  put 
the  division  name  in  parentheses.  A  master  listing  of  the  contractors’ 
names  and  addresses  is  maintained,  and  within  that  listing  cross 
references  are  made  (40;  50). 

At  Space  Systems  Division  the  original  report  is  filed  according  to 
parent  corporation  and  division/subsidiary  (26;  55). 

At  Munitions  Systems  Division,  the  focal  point  files  the  original  in 
a  file  folder  in  the  source  selection  vault  by  parent  corporation. 
Assessment  reports  are  not  filed  by  any  other  means;  however,  plans  are 
being  made  to  color  code  the  assessment  reports  by  year  and  also 
segregate  them  by  contractor  division  or  subsidiary  (9). 

At  Aeronautical  Systems  Division,  portions  of  the  assessment  report 
are  first,  entered  into  a  data  base  on  the  Wang  computer  system.  Fields 
include  a  nine-dig' t  reference  number,  the  parent  corporation,  contractor 
division  or  subsidiary,  address,  DODAAD,  period  covered  by  the  initial 
assessment,  report  and  the  latest  interim  or  final  report,  contract 
number,  two- letter  organizat ion  responsible  for  the  assessment.,  and  a 
brief  descript  ion. of  the  program.  The  focal  point  stated  that  it  takes 
approx  itiu.it  ely  two  minutes  to  enter  the  information  into  the  computerized 
data  base.  The  data  base  is  used  for  generating  reports  for  Air  Force 
Systems  Command  and  to  show  the  PRAG  members  uhieh  assessment  reports  are 
on  file  (17). 

The  Aeronautical  Systems  Division  focal  point  then  makes  copies  of 
the  report  and  files  the  original.  The  f oca i  point  files  the  forms  by 
parent  corporation,  contractor  division,  and  in  order  by  contract  number. 
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Ihe  nine  digit  reference  number  xxx-yyy- zzz  is  an  integral  part  of  the 
filing  system.  The  first  three  digits  (xxx)  correspond  to  the  parent 
corporation.  The  next  three  digits  (yyy)  correspond  to  a  division  within 
the  parent  corporation.  The  final  three  digits  show  the  sequential  order 
of  the  actual  contract  w'ithin  a  division/subsidiary.  The  parent 
corporation  number  corresponds  with  the  order  in  which  the  name  is 
presented  in  the  CPARS  regulation.  For  example  012-yyy-zzz  wd  1 1  be  the 
twelfth  contractor  listed  in  Air  Force  Systems  Command  Regulation  800-54. 
The  focal  point  was  not  sure  how  (or  if)  the  numbers  would  be  updated  if 
Air  Force  Systems  Command  added  more  contractors  to  the  list  (17). 

Several  focal  points  stated  that  there  were  attachments  other  than 
the  continuation  sheets  for  Blocks  16  and  18  which  may  accompany  the 
assessment  report  (26;  30;  40;  50;  55).  The  attachments  could  be  a 
letter  from  the  contractor  or  comments  made  by  the  reviewing  official  or 
program  manager.  In  all  cases  the  attachments  were  not  included  with  the 
copies  of  the  assessment  reports  sent  to  the  other  product  divisions.  At 
Electronic  and  Space  Systems  Divisions,  the  attachments  were  kept  with 
the  original  assessment  report  (26;  30;  55).  At  Ballistic  Systems 
Division,  a  separate  folder  has  been  established  to  file  any  working 
papers  or  other  attachment  which  support  the  evaluation  (40;  50). 

Maintenance .  Four  basic  maintenance  functions  were  identified 
by  the  CPARS  focal  points:  update  the  data  base  when  a  company  merges  or 
is  bought  out  by  another;  identify  assessment  reports  which  are  to  be 
retained  longer  than  five  years  by  direction  of  the  program  manager; 
remove  expired  assessment  reports;  and  maintain  correct  descript ive 
information  on  each  contractor  (name,  address,  points  of  contact). 
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Contractor  Name  Changes.  Two  methods  were  suggested  for 
handling'  the  merging  of  contractors.  The  first  method  involves  resorting 
all  existing  assessment  reports  into  a  new  folder.  Assessment  reports 
for  two  companies  which  may  merge  together  could  be  combined  under  the 
new  contractor  name.  Notes  would  then  be  made  of  the  name  change  so  that 
there  was  a  way  to  trace  the  assessment  reports  (11;  17;  30).  In  the 
second  method  suggested,  assessment  reports  would  remain  in  their 
original  folders,  and  each  folder  would  be  cross  referenced  to  the  other 
and  to  a  third  folder  with  the  new  contractor  name  (9;  26;  55). 

Obsolete  Assessment  Reports .  The  suggested  method  for 
removing  expired  reports  and  the  time  when  reports  actually  became 
obsolete  differed  among  the  product  divisions.  If  a  program  manager 
wanted  to  keep  a  particular  assessment  report  for  more  than  five  years 
then  notification  would  need  to  be  given  to  the  focal  point  (9;  26;  55). 
The  focal  point  would  make  a  record  of  the  conversation  and  put  a  cover 
letter  on  the  assessment  report  explaining  that  it  must  be  retained  (9). 

The  focal  point  from  Munitions  Systems  Division  plans  on  removing 
the  assessment  reports  that  are  five  years  old  and  microfiche  them  (9). 

Ballistic  Systems  Division  plans  on  maintaining  a  manual  suspense 
log.  When  the  final  assessment  report  is  written  the  contract  would  be 
written  in  a  suspense  log.  The  clock  would  then  be  started  and  all 
reports  for  that  contract  would  be  kept  for  five  years  (40;  50). 

At  Space  Systems  Division  the  five  year  period  is  counted  from  the 
date  of  the  reviewing  official’s  signature.  After  five  years  the  report 
will  be  removed  (26;  55).  Aeronautical  and  Electronic  Systems  Divisions 
have  not  made  plans  for  the  removal  of  obsolete  reports. 
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Update  of  Informal  ion.  Ballistic  Systems  Division 
maintains  a  computer  data  base  of  names  and  addresses  for  those 
contractors  on  which  they  originate  assessment  reports.  In  May  of  each 
year,  the  focal  point  sends  out  a  list  to  all  the  contractors  requesting 
confirmation  on  the  names,  addresses,  and  points  of  contact  for  which  the 
assessment  reports  should  be  delivered.  Each  year  they  find  that  there 
are  many  changes  which  need  to  be  made  (48;  58). 

Out  put .  The  manner  in  which  the  PRAG  members  obtain 
information  from  the  CP.VRS  differed  across  the  product  divisions. 
Differences  included  the  way  PRAG  members  requested  information,  access 
allowed  to  the  original  file  folders,  and  whether  copies  are  made  of  the 
assessment  reports. 

PRAG  members  at  Electronic  Systems  Division  must  send  a  letter 
requesting  access  to  the  CPARS  to  the  focal  point.  Only  members  named  in 
an  access  letter  may  see  the  assessment  reports  (11;  38). 

At  Electronic  and  Space  Systems  Divisions,  the  PRAG  members  give  the 
name  of  contractors,  programs,  and  contract  numbers  which  they  are 
interested  in  (11;  26;  38;  55).  Space  Systems  Division  has  developed  a 
form  so  that  the  focal  point  has  ail  the  information  available  to  search 
through  the  assessment  reports  (26;  55).  According  to  the  focal  po;nt  at 
Electronic  Systems  Division,  the  entire  folder  for  a  given  contractor  is 
not  usually  given  to  the  PRAG  member,  but  there  is  nothing  prohibiting 
the  PRAG  member  from  seeing  the  entire  folder  (11;  38). 

At  Electronic  and  Space  Systems  Divisions,  no  copies  of  the 
assessment  reports  are  made  to  support  the  PRAG  analysis.  The  original 
reports  are  given  to  the  PRAG  members,  who  then  take  them  to  the  area 
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which  they  are  working  on  the  source  selection  (11;  26;  38;  53).  PRAG 
members  are  only  given  access  to  the  contractor  files  which  they  are 
interested  in  (11;  3Q). 

At  Munitions  Systems  Division  PRAG  members  give  the  focal  point  the 
name  of  the  contractor  and  the  specific  division.  PRAGs  look  at  all 
assessment  reports  for  a  specific  division  and  then  for  the  entire 
company.  The  focal  point  does  not  perform  any  sorting  of  the  forms.  No 
copies  of  the  reports  are  made.  The1  PRAG  members  review  the  original 
copies  of  the  assessment  reports  in  the  source  selection  facility.  This 
has  not  been  a  problem  since  the  PRAG  members  are  working  in  the  same 
facility  when  performing  the  risk  assessment  (9). 

At  Aeronautical  Systems  Division  PRAG  members  look  at  a  list  which 
shows  all  the  reports  in  the  file.  The  list,  generated  by  the  Wang 
data  base,  shows  the  parent  corporation,  reference  number,  number  of 
intermediate  reports  and  final  report  if  applicable,  contract  number, 
contractor  division,  and  a  brief  description  of  the  program.  The  PRAG 
members  look  at  the  list  and  identify  which  assessment  reports  they  would 
like  to  see.  Copies  are  then  made  and  given  to  the  PRAG  member. 

According  to  the  Aeronautical  Systems  Division  focal  point,  the  PRAG 
members  do  not  have  access  to  the  original  files.  However,  if  a 
computerized  data  base  for  the  CP.ARS  could  be  developed  there  would  be  no 
problem  with  PRAG  members  searching  through  the  reports  as  long  as  the 
system  could  limit  access  to  only  the  contractor  of  interest  (17). 

At  ballistic  Systems  Division,  the  PRAG  members  ask  for  all  the 
assessment  reports  for  a  given  contractor,  and  also  look  at  specific 
divisions  within  a  company.  PRAG  members  sort  through  the  contractor 
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folder  for  relevant  assessment  reports,  and  make  copies.  PRAG  members 
are  allowed  to  take  them  back  to  the  source  selection  area  where  they  are 
working.  The  copies  of  the  assessment  reports  used  become  a  permanent 
part  of  the  source  selection  record  (40;  50). 

Reporting.  Each  person  interviewed  was  asked  what  reports  they 
need  to  generate  as  the  CPARS  focal  point. 

As  stated  earlier,  four  of  the  five  product  divisions  submit 
quarterly  or  more  frequently  to  Air  Force  Systems  Command  a  report 
describing  the  latest  assessment,  reports  (11;  17;  30;  40;  50).  These 
reports  contain  the  contractor  name,  subsidiary  of  division,  contract 
number,  period  or  performance,  and  a  brief  description  of  the  contract 
(11;  17;  30). 

In  addition,  Air  Force  Systems  Command  Regulat ion  900-54  (1:3) 
requires  each  focal  point  to  submit  annually  a  report  which  states  the 
name  and  address  of  the  contract  division  or  subsidiary;  the  parent 
corporation;  the  number  of  times  the  contractor  has  submitted  proposals; 
and  any  new  offerors  for  inclusion  in  the  data  base  (9;  11;  26;  30;  40; 
50;  55). 

The  focal  points  also  had  to  prepare  reports  for  local  use  within 
the  product  divisions. 

At  Electronic  Systems  Division  occasioially  the  focal  point  must 
prepare  a  talking  paper  stating  the  number  of  assessment  reports  on  file, 
the  number  of  contractors,  number  of  assessment  reports  per  contractor, 
and  the  number  of  programs  which  have  had  PRAGs  and  looked  at  the  CPARS 
(11;  30). 
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The  focal  point  at  Munitions  Systems  Division  builds  a  report  that 
goes  to  the  commander  on  incomplete  assessment  reports  from  informat  ion 
gathered  while  tracking  their  progress.  One  chart  is  prepared  which  has 
a  list  of  contracts,  the  date  assessment  reports  were  due  out,  the 
actually  mailing  date,  the  date  reports  were  returned  by  the  contractor 
and  the  date  of  approval  by  the  reviewing  official  (9). 

Prob 1 ems  Inherent  to  the  Process .  Each  focal  point  was  asked  if 
they  had  found  any  problems  inherent  to  this  process  and  whether  they  had 
any  suggestions  for  recommended  improvements. 

Ms.  Irene  Biddy  from  Electronic  Systems  Division  stated  that  the 
only  problems  were  with  educating  the  people  and  getting  the  process 
st  art ed  (11). 

Ms.  Gail  Yranicar  from  Space  Systems  Division  stated  that  there  is  a 
mandate  to  streamline  the  process;  reduce  the  number  of  evaluators  and 
the  time  fo~  evaluation.  At  the  same  time,  there  is  the  need  to  have  a 
PR\G.  To  keep  the  number  of  people  down,  Space  Systems  Division  often 
makes  members  of  the  evaluation  team  perform  double  duty  with  the  PRAG, 
which  stretches  out  the  time.  Dedicating  people  to  the  PRAG  function 
also  happens,  but  at  a  cost  of  increased  personnel  requ i rements .  Ms. 
Yranicar  emphasized  that  the  PRAG  process  was  added  value  to  the 
selection  decision  and  is  worth  the  extra  effort  (55). 

A  second  problem  identified  by  Ms.  War.  i  car  was  that  prior  f  o  the 
CP\RS,  Space  Systems  Division  generated  detailed  ;^uest.  ionnaires  tailored 
to  the  specific  programs .  In  terms  of  relevancy,  Ms.  Yranicar  was  not 
sure  tliat  an  assessment  report  from  CPARS  could  provide  the  necessary 
informat  ion  without  follow-up  interviews.  She  stated  that  the  (PARS  was 
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great  in  that  there  is  a  standi ng  uody  of  data  and  DRAG  members  have  some 
place  t o  start  ( 55 ) . 

The  only  problem  identified  with  the  process  by  Mr.  Michael  Cushing 
of  Aeronautical  Systems  Division  was  that  there  have  been  requests  to  use 
the  CP.ARS  information  for  other  than  source  selections.  Air  Force  Audit 
.Agency,  system  program  managers,  and  Air  Force  Systems  Command  have  asked 
for  copies  of  assessment  reports  (17). 

A  problem  identified  by  Mr.  David  Smith  from  Ballistic  Systems 
Division  involves  the  identific  .tion  of  the  corporate  chief  executive 
officer  or  authorized  person  for  a  company  who  must  submit  letters 
requesting  authorization  access  to  review  assessment  reports  in  the 
CP.ARS  (50).  The  guidance  provided  in  Air  Force  Systems  Command 
Regula*  ion  8QQ-54 ,  paragraph  O.b,  (1:3-4)  states  that  a  CP.ARS  access 
letter  from  the  corporate  chief  executive  officer,  or  a  letter  by  the 
corporate  chief  executive  officer  designating  other  approval  officials, 
in  addition  to  a  CPARS  access  letter  signed  by  the  designated  person  must 
be  presented  to  the  CP.ARS  focal  point  .  The  basic  quest  ion  is  whether  the 
initial  corporate  official  must  be  from  he  parent  corporation  or  whether 
the  official  can  be  the  chief  executive  of  the  division  or  subsidiary. 

At  Ballistic  Systems  Division  each  assessment  report  is  being 
entered  on  a  word  processing  template  by  the  administrative  assistant. 

Ms.  Pat  Olson,  the  CP.ARS  administrative  assistant  from  Ballistic  Systems 
Division,  stated  that.  Ballistic  Systems  Division  is  approx imate ly  one 
third  through  this  cycle  and  it  has  been  very  labor  intensive  so  far. 

Ms.  Olson  also  stated  that  she  expects  the  worst,  is  yet  to  come  (50). 
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Mr.  George  Bates,  the  CPARS  focal  point  for  Munitions  Systems 
Division,  identified  several  problems  with  the  current  process.  Two 
problems  are  the  reluctance  of  program  managers  to  complete  «n  assessment 
report,  and  getting  the  assessment  report  mailed  to  the  contractor  on 
t  ime . 

For  program  managers,  preparing  the  assessment  report  is  a  low- 
priority  item  when  compared  to  their  other  functions.  Also  the  two- 
letter  officers  are  busy  people  and  review  of  the  assessment  report 
takes  time.  (9) 

The  real  problem,  according  to  Mr.  {kites,  occurs  once  the  assessment 
report  has  been  returned  by  the  contractor.  The  wait  between  the  time 
the  two- let  ter  receives  the  assessment  report  and  the  point  where  the 
reviewing  official  signs  it  is  unacceptable.  Mr.  Bates  stated  that  there 
are  eight  different  coordination  blocks  to  get  a  single  assessment  report 
to  the  reviewing  official  (9). 

Protect  ion  of  Information.  According  to  the  CPARS  regulation,  all 
information  contained  in  the  system  is  considered  source  selection 
sensitive.  Each  focal  point  was  asked  what  special  constraints,  for  the 
transportation,  protection,  and  storage  of  this  data  was  currently  being 
exerc i sed . 

Tin*  Air  Force  Systems  Command  Form  125  is  marked  "Source  Selection 
Sensitive  --  For  Official  Use  Only  (when  filled  out)"  (17).  According  to 
the  Aeronautical  Systems  Division  focal  point,  there  could  be  several 
different  interpretations  as  to  when  the  form  is  actually  filled  out  and 
becomes  source  selection  sensitive.  It  could  be  when  the  program  manager 
provides  the  initial  evaluation  or  when  the  reviewing  official  finally 
signs  the  form  in  Block  21.  Once  it  is  filled  out  and  gets  to  the  focal 
point,  it  is  protected  at  all  times.  There  is  no  instructions  in  the 


-4-51 


current  supplement  which  describe  how  to  handle  the  assessment  report 
prior  to  it  being'  completed  (17). 

The  assessment  reports  are  kept  in  a  file  drawer  in  a  locked  room  or 
vault  with  controlled  access  and  are  only  released  to  PRAG  members  with 
authorized  access  (9;  11;  26;  40;  58;  55).  At  Munitions  Systems  Division 
no  one  can  get  to  the  CPARS  unless  they  deal  directly  with  the  CPARS 
focal  point  (9).  While  in  transit  between  offices,  the  assessment  report 
is  hand  carried  and  kept  underneath  a  source  selection  sensitive  cover 
sheet.  The  assessment  report  is  never  left  unat tended  (9;  11;  26;  48; 

58;  55).  When  assessment  reports  are  mailed,  they  are  double  wrapped  and 
appropriately  marked  (11;  26;  48;  58;  55).  The  transmittal  letter  to  the 
contractor  highlights  the  fact  that  the  report  is  source  selection 
sensitive  and  the  restrictions  on  its  use  (26;  55). 

With  the  exception  of  Ballistic  Systems  Division,  the  assessmeni 
reports  are  not  prepared  using  word  processors  but  are  being  typed  on 
r ypcwr i t ers .  Ms.  Irene  Biddy,  from  Electronic  Systems  Division,  stated 
that  if  the  program  managers  uere  to  use  word  processors  there  would  have 
to  be  some  special  type  of  protection  constraints.  She  further  stated 
that  this  may  require  the  implementation  of  an  automated  CPARS  to  use  a 
full  time  data  input  clerk  whose  responsibility  would  be  to  key  in 
assessment  reports  (11). 

Sizing  the  CPARS  Data  Base.  Focal  points  were  asked  to  identify  the 
ruimber  of  assessment  reports  which  originated  from  their  product  division 
and  t o  estimate  how  that  number  would  increase  based  on  new  contracts  and 
addition  of  new  contractors.  The  existing  number  of  assessment  reports 
for  each  product  division  is  listed  in  Table  4-6. 
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Table  4-6 


Number  of  Assessment  Reports  Currently  in  the  CP.\RS 


Product  Division 
Aeronautical  Systems  Division 
Ballistic  Systems  Division 
Elect  ron i c  Sys  t  ems  Division 
Munitions  Systems  Division 
Space  Sys  t  ems  D i v i s i on 


Number  of  Assessment  Reports 
138 
58 
63 
34 
47 
324 


Focal  points  from  Aeronautical,  Ballistic,  Electronic  and  Space 
Systems  Divisions  expect  an  increase  in  over  five  times  the  amount  of 
assessment  reports  currently  in  the  system  in  the  next  five  years. 
However,  the  focal  points  felt  that  the  increase  would  not  be 
subs t ant ial ly  more  than  five  times  the  amount  unless  the  system  expands 
to  include  service  contracts  (11;  17;  26;  48;  58;  55). 

These  same  focal  points  recognize  that  there  has  been  some  interest 
to  add  other  contractors  to  the  list.  However,  they  do  not  anticipate 
the  number  of  contractors  increasing  unless  the  CPARS  is  expanded  to 
include  other  type  contracts  (11;  17;  26;  48;  58;  55). 

Mr.  George  Bates  from  Muni t ions  Systems  Division  anticipates  the 
number  of  reports  to  grow  much  larger  than  five  times  the  number  of 
reports  in  the  system  (9).  Mr.  Bates  also  expects  the  number  of  players 
to  increase  dramatically.  Mr.  Bates  stated  that  he  has  about  eight  new 
companies  which  Munitions  Systems  Division  would  like  to  add  to  the  list. 
Each  year  he  expects  that  they  would  add  more  names  (9). 
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If  there  were  a  ten-percent  increase  in  the  number  of  contracts  each 
year  (which  seems  appropriate  and  also  conservative  considering  the 
information  provided  by  the  focal  points)  the  number  of  reports  which 
could  be  expected  in  the  system  is  shown  in  Table  4-7.  The  figures 
shown  in  Table  4-7  are  based  on  the  assumption  that  reports  would  be  held 
for  a  full  five  years  and  that  a  negligible  number  of  contracts  would  be 
completed  during  that  period.  It  is  expected  that  sometime  after  the 
initial  five  years,  the  number  of  initial  assessment  reports  written  on 
new  contracts  would  not  exceed  the  number  of  contracts  expiring  for  a 
given  period.  At  that  point  in  time  the  number  of  contracts  in  the 
system  would  remain  fairly  constant. 

Table  4-7 
Growth  of  CPARS 


1989 

1990 

1991 

1992 

1993 

1994 

Number  of  Added 
Assessment  Reports 

324 

356 

392 

431 

474 

521 

Total  Number  of 
Assessment  Reports 

324 

680 

1072 

1503 

1977 

2498 

'Interim  or  final  reports  plus  initial  reports  for  new?  contracts 


Existing  Computer  Systems .  Each  focal  point  was  asked  to  identify 
the  computer  systems  and  software  packages  currently  in  use  in  their 
offices.  Their  response  is  summarized  in  Table  4-8.  The  small  computer 
technical  centers  at  Eglin  and  Hanscom  Air  Force  Bases  were  contacted  to 
determine  whether  Zenith  Z-248  or  other  MS-DOS  based  computers  were 
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currently  in  use.  All  five  product  division  locations  currently  use  and 
support  the  7,-248  computer,  although  not  all  focal  points  have  access  to 
Z-248  computers. 


Table  4-8 


Existing1  Computer  Systems 


Produc  t  Division 

Computer  Systems 

Software  Packages 

Electronic  Systems 

Wang 

— 

1  CPT 

-- 

Space  Systems 

Z-248s 

WordSTAR 

Lotus  1-2-3  R 

Mu  1 1  iMat.e 

Eiiable™ 

Aeronaut ical  Systems 

Z-248s 

-- 

Wang 

-- 

Ballistic  Systems 

Z-248s 

WordPerfect ' B 1 

Lotus  1-2-3 
dBASE  1 1 1  R  1 

Harvard  Graphics™ 

Munitions  Systems 

2 

-- 

1  Only  the  CPT  is  used  for  CPARS 
computers  are  currently  in  use 

related  information.  Z-248 
at  Hanscom  Air  Force  Base. 

*  Does  not  have  any  computers  as 
computers  are  current ly  in  use 

the  CPARS  focal  point.  Z-248 
at  Eglin  Air  Force  Base. 

Capab i 1 i t i es  Expec t ed  From  an  Automated  System.  The  focal  points 
were  asked  to  think  about  the  performance  assessment  process  and  explain 
how  an  automated  information  system  would  apply  to  each  activity.  The 
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activities  are  listed  below  with  the  focal  points  suggested  capabilities. 
Suggestions  were  made  for  six  of  the  seven  activities. 

Data  Col  lection.  Electronic  Systems  Division  keeps  a  suspense 
file  so  that  the  program  manager  can  be  given  75  days  before  an 
assessment  report  is  due.  This  provides  36  days  for  preparation,  3® 
days  for  a  contractor  to  respond,  and  15  days  for  review  and  approval. 

Ms.  Irene  Biddy  and  Ms.  Cynthia  Keefe  stated  that  it  would  be  nice  if  a 
computerized  system  provide  that  capability.  They  also  stated  that  there 
would  have  to  be  a  manual  backup  because  the  system  that  they  have  is 
notorious  for  being  down  (11;  30). 

According  to  Mr.  Michael  Cushing,  an  automated  system  would  help  the 
focal  point  and  the  PRAG  members,  but  would  probably  make  life  for  the 
program  offices  more  difficult  (17).  Mr.  Cushing  stated  that  it  would  be 
up  to  the  program  offices  to  input  the  data.  He  anticipates  the  program 
manager  would  provide  a  disk  with  everything  on  it.  The  disk  could  then 
be  put  on  the  Z-248  to  update  the  data  base  (17). 

Data  Input .  Mr.  Cushing  stated  that  he  would  expect  to  receive 
a  disk  with  an  entire  assessment  report  on  it.  He  wanted  to  make  it 
clear  that  the  source  selection  office  at  Aeronautical  Systems  Division 
did  not  have  the  resources  to  enter  the  entire  assessment  report  into  a 
computer  data  base.  Mr.  Cushing  also  stated  that  the  time  to  get  the 
whole  assessment  report  into  the  data  base  should  not  exceed  ten  minutes 
(17).  Mr.  Cushing  currently  spends  between  two  and  three  minutes  to 
update  the  Wang  data  base  when  each  assessment  report  is  received  (17). 

Ballistic  Systems  Division  is  already  using  the  Acquisition 
Management  Information  System  and  the  data  base  of  contractor  names  and 
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addresses  to  help  the  initiation  of  the  process.  Mr.  Smith  and  Ms.  Olson 
also  stated  that  the  word  processing'  template  created  using  WordPerfect 
or  something  similar  could  be  used  for  data  input  (40;  50). 

Maintenance.  No  capabilities  were  suggested  for  this  activity. 

.Analysis.  According  to  Mr.  Smith,  a  sorting  capability  to  help 
the  PRAG  members  find  relevant  reports  would  be  a  big  benefit  (50).  Mr. 
Smith  also  suggested  that  if  there  was  a  central  data  base,  then  one 
office  could  look  at  the  reports  and  come  up  with  a  corporate  trend. 

This  would  then  provide  PRAGs  across  all  the  product  divisions  a 
consistent  corporate  profile  for  each  contractor  (50). 

Output .  .An  automated  system  could  help  with  the  quality  of  the 
reports.  With  reproduction  of  the  forms,  there  is  an  opportunity  to  get 
poor  copies  (26;  55).  .An  automated  system  could  help  with  transmission 
of  the  reports.  Space  Systems  Division  has  people  working  source 
selections  in  Florida  and  throughout  the  state  of  California.  Under  the 
current  system,  it  is  difficult  to  get  the  information  to  these  people. 
For  the  source  selection  in  Florida  the  focal  point  needed  to  make  copies 
of  the  assessment  reports,  which  was  against  Space  Systems  Division 
policy  (26;  55).  The  capability  to  download  information  directly  from 
the  assessment  reports  to  the  charts  that  need  to  be  briefed  to  the 
Source  Selection  Authority  would  also  be  helpful  (40;  50). 

Report ing.  All  five  focal  points  and  their  assistants  felt 
that  an  automated  system  could  very  easily  generate  the  types  of  reports 
which  must  be  provided  by  the  CPARS  focal  point. 

Protect  ion.  The  focal  points  stated  that  the  system  must  be 
able  to  provide  protect ion  of  the  data  in  such  a  way  that  a  PRAG  member 
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or  contractor  could  be  limited  access  to  a  specific  contractor.  There 
needs  to  be  some  way  to  segregate  the  data  base  so  that  access  can  be 
controlled  (11;  38). 

User  of  the  Automated  System.  Given  that  an  automated  information 
system  could  be  developed  to  help  collect,  process,  and  disseminate 
contractor  performance  information  to  assist  PRAGs  in  their  performance 
risk  assessment  each  focal  point  was  asked  who  would  be  the  appropriate 
person  to  sit  in  front  of  the  computer  terminal  and  generate  the  product. 

At  Aeronautical,  Ballistic,  Munitions  and  Space  Systems  Divisions, 
the  focal  points  felt  that  the  PRAG  members  would  be  the  best  users  of 
the  system  (9;  17;  26;  40;  50;  55).  Mr.  Smith  from  Ballistic  Systems 
Division  stated  that  he  could  not  see  any  benefit  in  having  a 
representative  from  the  focal  point's  office  perform  the  sorts. 

The  PRAG  members  know  what  is  relevant  and  so  it  would  be  in 
their  best  interest  to  sit  down  at  the  system.  I  believe  that  it 
would  save  a  lot  of  time  if  the  reports  could  be  sorted  through  in 
some  automated  fashion.  (50) 

Ms.  Keefe  from  the  Electronic  Systems  Division  felt  that  a 
representative  from  the  CPARS  Focal  Point's  office  would  be  the  most 
efficient  user  of  the  system. 

This  way  only  one  person  would  be  going'  into  the  system.  It 
wo\ 1 1 d  require  a  change  of  duties  for  the  administrative  assistant, 
but  it  would  result  in  the  most  efficient  way  of  doing  business. 

The  PRAG  could  provide  instruct  ions  and  the  assistant  could  create 
an  information  product.  (38) 

Ms.  Keefe  stated  that  she  did  not  know  of  any  reason  why  a  PRAG 
member  could  not  perform  the  sorts  and  build  reports  themselves  as  long 
as  the  data  could  be  protected  to  limit  access  to  only  contractors  for 
the  instant  source  selection  (11).  This  same  concern  for  isolation  of 
the  data  was  expressed  by  individuals  from  Aeronaut ical  and  Space  Systems 
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Divisions  (17;  26;  55).  Mr.  Cushing  from  Aeronautical  Systems  Division 
also  felt  tl\at  the  focal  point  should  control  any  printing  of  CPAftS  datct 
by  the  FRAG  members  (17). 

Development  of  the  Requirements  Basel ine 

The  requirements  baseline  represents  a  synthesis  of  all  requirements 
determined  to  this  point.  The  capabilities  expressed  during  the 
interviews  with  PRAG  members  and  CPARS  focal  points,  the  procedures 
necessary  to  support  the  PEAG  process  and  CPARS  administration  functions, 
and  the  regulatory  constraints  applied  to  the  information  and  to  the 
system  all  had  to  be  consolidated  into  a  concise  set  of  system  levei 
requirements.  The  development  of  the  requ i remen t s  baseline  implements 
the  first  part  of  the  final  objective  of  this  research  project. 

Objective  4.  Establish  the  requirements  baseline  and  conceptual 
design  for  an  automated  information  system,  based  on  the  CPARS,  to 
collect,  process,  protect,  and  disseminate  contractor  performance 
assessments. 

Investigative  Question  4.  What  system  capabilities  are  dictated  by 
the  information  requirements  of  PRAGs  and  the  implementation 
procedures  of  the  CPARS? 

The  requirements  baseline  represents  the  system  level  capabilities 
which  must  exist  in  order  to  meet  the  user’s  needs.  The  baseline  will 
include  both  expressed  and  derived  requirements. 

Expressed  requirements  are  the  specific  attributes  of  the  system 
which  have  been  stated  by  the  user.  Tn  this  case,  the  expressed 
requirements  are  the  system  capabilities  which  have  been  identified  by 
the  PR^G  representatives  and  the  CPARS  focal  points. 

Derived  requirements  represent  those  which  have  not  been  explicitly 
stated,  but  are  essential  to  make  the  system  operable.  The  primary 
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sources  for  the  derived  requirements  are  the  procedures  used  and  types  of 
information  required  by  past  PRAGs,  and  the  administrative  procedures  in 
effect  to  collect,  process,  and  distribute  contractor  performance 
information.  Lover  level  derived  requirements  will  also  appear  during 
the  design  process  as  specific  interfaces  are  identified. 

Each  detailed  step,  activity,  and  desired  capability  as  described  by 
every  one  of  the  PRAG  members  and  GPARS  focal  points  was  carefully 
analyzed  to  extract  the  system  level  requi rements .  The  intent  of  this 
process  was  to  identify  all  requirements  independent  of  the  number  of 
individuals  who  recognized  them.  A  common  thread  through  the  process  was 
sought  to  maintain  coherence  and  consistency.  No  effort  was  made  to 
p r i or i t i ze  requ i remen t s . 

The  primary  reason  for  developing  the  requirements  baseline  was  to 
support  the  construction  of  the  conceptual  model  of  the  system.  The 
conceptual  model  replicates  much  of  the  content  of  the  requirements 
baseline  while  providing  a  user  orientation  to  the  system  design.  Since 
the  conceptual  design  will  be  described  in  detail,  the  requirements 
baseline  has  been  presented  as  an  appendix.  The  interested  reader  may 
want  to  review  Appendix  E:  Requ i rements  Basel ine  prior  to  Chapter  V  in 
order  to  follow  the  complete  process  of  this  project. 
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V .  Deve  i  opment/Des  i  gn 


Concep t  ua 1  Des ign 

The  conceptual  design  is  a  user  oriented  description  of  the  required 
capabilities  of  the  system.  The  conceptual  design  describes  the 
applications  of  the  system  as  the  user  sees  them  (18:577).  The  basis  of 
the  conceptual  model  was  established  early  during  the  requirements 
analysis  stage  with  the  characterization  of  the  performance  assessment 
process  as  seven  different  activities.  These  activities  are  data 
collection,  data  input,  maintenance,  analysis,  output,  reporting,  and 
protection.  The  purpose  of  this  section  is  to  describe  the  relat ionships 
among  the  seven  activities,  and  the  interaction  of  the  activities  with 
the  core  of  the  system,  the  data  base  of  performance  assessment  reports. 

Investigative  Question  5.  How  do  the  established  requirements 

affect  the  architecture  for  the  automated  information  system? 

To  answer  this  question,  the  expressed  and  derived  requirements 
specified  in  the  requirements  baseline  had  to  be  allocated  to  the 
architecture  of  t he  automated  system.  A  transactional  model  as  described 
by  Dr.  Roger  Pressmen  in  So  f  tjware  Engj  nee  ring,  a  Pract  i  t  i  oner’s  Approach, 
was  chosen  for  this  purpose  (17:202-265).  The  model  consists  of  a  hub 
or  decision  center  which  can  start  one  or  more  of  the  many  different 
transactions  required  by  the  system.  Bach  of  the  requirements 
established  by  the  user  needs  to  he  allocated  to  one  or  more  transactions 
within  the  system. 

Top-1  eve !  Conceptual  Design.  The  top-level  transactional  model  for 
the  CPARS  is  shown  in  Figure  5-1.  The  transaction  center  acts  as  the 


Figure  5-1.  Top  Level  Transaction  Flow  Diagram  For  CPARS  Automation 


interface  to  the  external  world  for  all  applications  of  the  system.  The 
hub  also  acts  as  a  protection  mechanism  for  the  sensitive  performance 
information.  In  much  the  same  way  as  the  focal  point  restricts  access  to 


the  files  of  assessment  reports,  the  transaction  center  will  control  the 
interface  to  the  CPARS  data  base. 

The  user,  through  some  interface  to  the  system,  will  select  one  of 
many  different  transactions.  This  is  portrayed  as  a  menu  in  the 
transaction  flow  diagrams.  The  system  will  perform  requested  functions 
based  on  the  access  allowed  to  the  particular  individual.  Access  to  the 
system  would  be  defined  and  controlled  by  the  CPARS  focal  point.  Because 
protection  of  the  sensitive  information  must  be  an  inherent  part  of  every 
action  in  the  system,  it  is  not  depicted  as  a  separate  activity. 

Lower- level  Conceptual  Des ign.  The  following  series  of  transaction 
flow  diagrams  allocate  the  requirements  stated  in  the  baseline  to  the 
conceptual  design.  In  many  instances,  the  requirements  overlap  more  than 
one  of  the  seven  activities  and  the  boundaries  tend  to  blur.  This  is 
especially  true  for  data  collection  and  data  input,  analysis  and  output, 
and  output  and  reporting  activities.  Four  sunmary  transaction  flow 
diagrams  arid  eleven  detailed  transactional  flow  diagrams  will  be 
discussed  in  the  remainder  of  this  section. 

Perform  Data  Col lect ion  and  Data  Input  Act ivi t ies .  Due  to  the 
close  relationship  of  the  data  collection  and  data  input  activities 
they  have  been  combined  and  will  be  described  together.  The  summary 
transaction  flow  diagram  for  data  collection  and  input  is  shown  in 
Figure  5-2. 

The  statement  in  the  top  right  corner  of  the  diagrrim  represents  an 
option  from  the  main  menu  for  the  CPARS  automated  system.  The  seven 
transactions  shown  beneath  the  center  represent  opt  ions  from  a  lower 
level  menu  for  data  collection  and  input. 
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User  Interface  Perform  Data  Collection  and 


Figure  5-2.  Data  Collection  and  Input  Summary  Transaction  Flow  Diagram 
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I dent i f y  Contracts  Wh i ch  Require  Ini t iai  and  Int ermediate 
or  Final  Assessment  Reports .  The  first  two  options  for  the  data 
collection  and  input  tasks  are  to  initiate  the  assessment  process.  The 
transactional  flow  diagrams  are  depicted  In  Figures  5-3  and  5-4. 

The  user  of  the  system  for  these  transactions  is  the  CPARS  focal 
point  or  the  two- letter  point  of  contact.  Initiation  processes  of 
selecting  contracts  for  initial  assessments  (Figure  5-3)  and  intermediate 
or  final  assessments  (Figure  5-4)  are  nearly  identical.  The  only 
differences  are  the  source  which  is  checked  to  identify  contracts  to 
report  on  for  a  given  period  and  the  amount  of  preparation  necessary  to 
create  new  records.  The  Acquisition  Management  Information  System  (AMIS) 
ur  a  similar  source  should  be  checked  to  identify  contracts  for  initial 
cissessment  s . 

Once  an  assessment  has  been  written  for  a  given  contract,  the  CPARS 
data  base  can  be  checked  for  interim  and  final  assessment  reports.  The 
CPARS  focal  point  will  use  information  extracted  from  the  Acquisition 
Management  Information  System  or  the  CPARS  data  base  to  create  new 
records  for  each  of  the  contracts  identified  and  then  notify  the  program 
manager  that  an  assessment  is  due.  When  initializing  the  record,  the 
name/address  data  base  should  be  checked  for  corrections  (see  the 
maintenance  transaction  flow  diagram  for  further  information  on  this  data 
base).  When  the  program  manager  is  notified,  the  focal  point  will  make 
an  ent ry  into  the  tracking  log. 
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CPARS  Identify  contracts  which 

Focal  Point  require  an  assessment  report 


Figure  5-3.  Identify  Contracts  Requiring  Initial  Assessment  Reports 
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CPARS  Identify  contracts  which 

Focal  Point  require  intermediate  or  final 


Figure  5-4.  Identify  Contracts  Requiring  Intermediate  or  Final 

Assessment  Reports 
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Perform  Initial  Evaluation.  Once  notified  that  an 


assessment  is  due,  the  program  manager  selects  the  "perform  initial 
evaluation"  option  frcm  the  data  collection  and  input  menu.  The 
transaction  flow  diagram  for  this  process  is  shown  in  Figure  5-5.  The 
program  manager  should  have  the  option  to  key  the  information  directly 
into  the  data  base  input  form  or  to  enter  into  an  interactive  environment 
with  the  system. 

When  program  managers  choose  the  interactive  environment,  they  will 
be  asked  probing  questions  for  each  item  in  Blocks  li  and  15  of  the 
report.  The  purpose  of  the  questions  is  to  help  the  program  manager 
provide  the  level  of  detail  >n  the  narrative  section  required  to  support 
the  PRAGs.  The  program  manager’s  answers  to  the  probing  questions  will 
be  annotated  in  the  portion  of  the  record  reserved  for  the  narrative 
section  (Block  16)  of  the  CPARS  form.  Examples  of  standard  language  or 
key  words  to  describe  the  phase  of  the  process,  type  of  technology,  and 
type  of  work  performed  should  be  available  to  the  program  manager.  Upon 
completing  the  assessment,  the  program  manager  would  enter  into  a  word 
processing  function  and  edit  the  comments.  The  program  manager  should  be 
able  to  review  any  prior  period's  assessment  report  for  that  contract. 

Once  the  evaluation  is  complete,  the  record  in  the  data  base  is 
updated.  A  transmittal  let  1 er  is  prepared  by  either  the  CPARS  focal 
point  or  t  -  ho  program  manager.  The  assessment  report  apd  letter  are  then 
readied  to  be  sent  to  the  contractor.  At  any  time  prior  to  the 
assessment  report’s  being  sent  to  the  contractor,  the  focal  point  should 
have  the  option  for  review. 
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Figure  5-5.  Perform  an  Evaluation  of  the  Contractor’s  Performance 
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Contractor ' s  Comments .  The  program  manager  or  CPARS  focal 
point  sends  the  assessment  and  transmittal  letter  to  the  contractor  by 
certified  mail  or  some  other  method.  When  the  contractor  receives  the 
assessment,  the  tracking  log  is  updated  and  the  36-day  clock  for 
contractor  comments  is  set.  Upon  receipt  of  the  response  from  the 
contractor  or  36  days  after  receipt  by  the  contractor,  the  focal  point 
will  update  the  tracking  log.  The  record  containing  the  preliminary 
assessment  is  updated  with  the  contractor’s  comments,  at  which  time  the 
program  manager’s  review  and  revision  process  begins.  The  transaction 
flow  diagram  for  this  phase  of  the  data  collection/data  input  process  is 
shown  in  Figure  5-6. 

Review  and  Revise  Assessment .  The  transaction  flow 
diagram  for  this  activity  is  presented  in  Figure  5-7.  The  program 
manager  reviews  the  assessment  report  and  determines  if  revisions  are 
necessary.  If  revisions  are  not  necessary,  the  report  is  validated  and 
submitted  to  the  product  division  reviewing  official. 

If  revisions  are  necessary,  the  information  in  Blocks  1  through  5  of 
the  assessment  report  is  transferred  to  a  new  record  and  the  statement 
"Revision  to  CPAR  for  period  dd/mm/yy  to  dd/mm/yy"  is  entered  in  Block  12 
of  the  new  record.  Rating's  or  coirenents  requiring  revision  are  then 
updated.  Reasons  for  the  changes  are  entered  in  the  remarks  section. 

The  program  manager  or  other  responsible  party  validates  the  assessment 
find  submits  it  to  the  reviewing  official.  When  action  is  complete  the 
tracking  log  for  that  contract  is  updated. 
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Program  Manager  or  Transmit  letter  and  assessment 

CPARS  Focal  Point  to  the  contractor,  and  receive 


Figure  5-6.  Transmit  Assessment  and  Receive  Contractor  Comments 
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Program  Review  and  revise  the  assessment 

Manager  based  on  the  contractor’s  comments . 


Figure  5-7.  Review  and  Revise  Assessment 
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Product  Division  Review.  The  product  division  reviewing 
official  examines  the  assessment  report  and  makes  any  correnents  in  Block 
20.  The  assessment  report  is  then  validated  by  signature  in  Block  21  or 
by  some  other  method.  If  the  review  is  performed  on-line,  the  record  can 
be  sealed  as  part  of  the  validation  process  so  that  it  is  forever  write 
protected.  If  a  hard  copy  is  presented  to  the  reviewing  official,  the 
sealing  step  must  be  performed  as  part  of  updating  the  local  and  command 
wide  data  bases.  The  transaction  flow  diagram  is  shown  in  Figure  5-8. 


Reviewing  Review  and  validate  the 

Official  performance  assessment  report. 


Figure  5-8.  Product  Division  Review 
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Update  Local  and  Command  Wide  Data  Bases .  If  not 
accomplished  during  the  previous  transaction,  the  reviewing  official’s 
conroents  must  be  inserted  into  the  record.  The  record  should  then  be 
sealed  so  that  it  cannot  be  altered  in  any  manner.  A  sealed  record  will 
permit  read  access  only.  The  record  will  then  be  inserted  into  the  local 
arid  command  wide  data  bases.  After  each  assessment  is  inserted  into  the 
CPARS  data  base,  or  at  some  regularly  scheduled  interval,  the  focal  point 
will  prepare  a  report  of  assessments  added  and  submit  it  to  Air  Force 
Systems  Command.  This  activity  is  depicted  Figure  5-9. 


CPARS  Update  local  and  coimnand-wide  data  bases. 

Focal  Point  _ _ _ 


Figure  5-9.  Update  CPARS  Data  Base 
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Maintenance  Activities.  The  user  for  this  function  is  the 
CPARS  focal  point.  There  are  four  primary  transactions  for  the 
maintenance  activity  as  depicted  in  Figure  5-1Q. 


CPARS 

Focal  Point 


Per Form  maintenance  activities. 


Menu 


CPARS 
Data  Bas 


Transact  ion 
Center 


Adjust  Data  Base 
for  Mergers  and 
Name  Changes 


Cross  reference 
Old  and  New 
Names 


Maintain  Correct 
Contractor 
Information 
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The  system  should  provide  the  capability  required  to  adjust  for 
mergers,  or  contractor  name  changes.  By  choosing  this  option  the  CPARS 
focal  point  should  be  able  to  cross  reference  records  in  the  data  base 
between  the  old  contractor  names  and  the  new  contractor  names.  Once 
accompl ished,  a  user  of  the  system  will  be  able  to  query  for  an 
assessment  report  on  a  contractor  and  not  have  to  worry  what  name  it  is 
listed  under  in  the  data  base. 

The  next  two  options  provide  the  focal  point  the  ability  to  search 
for  and  remove  expired  reports,  and  to  annotate  and  save  assessment 
reports  which  have  been  selected  by  program  managers  to  keep  for  longer 
than  five  years. 

The  final  maintenance  option  is  for  maintaining  a  data  base  of 
correct  names,  addresses  and  points  of  contact  for  contracts  in  which  a 
final  assessment  report  has  not  been  written.  Each  product  div.  'on 
would  track  the  contractors  for  which  they  originate  assessment  reports. 
Once  each  year,  the  CPARS  focal  point,  would  send  out  a  list  to  all  the 
contractors  request itg  confirmation  of  the  information.  When  corrections 
are  received  the  name/address  data  base  is  updated.  The  information  in 
this  data  base  can  be  checked  to  make  sure  records  for  intermediate  and 
final  assessments  are  correct  when  initiated. 

Ana lysis  Activities.  The  primary  users  for  the  analysis 
activities  are  the  PRAG  members.  The  CPARS  focal  point  will  have  to 
prepare  the  data  base  so  that  access  by  the  PRAG  members  is  limited  to 
the  contractors  for  the  immediate  source  selection.  The  summary- 
transaction  flow  diagram  for  analysis  activities  is  shown  in  Figure  5-11. 
Each  of  the  four  transactions  is  presented  in  lower  level  diagrams. 
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PRAG  Member 


Perform  ana  lysis  activities. 


Figure  5-11.  Analysis  Activities 

P 1  ann i rig  and  Documentation  Support .  The  PRAG  members  and 
the  program  manager  will  be  able  to  access  the  system  for  information  on 
regulations,  instructions,  local  supplements,  sample  formats,  and  word 
processing  templates  to  help  establish  the  responsibilities  and 
methodology;  and  make  sure  correct  words  and  format  are  used  for  required 
documentat ion. 

PRAG  members  can  use  the  system  to  review  frequently  used  criteria 
for  determining  relevancy.  The  PRAG  member  will  choose  the  criteria 
based  on  the  common  examples  and  any  unique  criteria  for  that  program. 

The  criteria  chosen  will  be  available  for  screening  examples  of 
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performance  for  relevancy,  and  determining  the  impact  of  specific 
examples  on  the  performance  risk  assessment .  The  transaction  flov 
diagram  for  planning  and  documentation  support  is  shown  in  Figure  5-12. 


FRAG  Member  Planning ‘  and  documentation  support. 


Figure  5-12.  fMann i ng  mid  Documentation  Support 
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Provide  Descript ive  Stat ist ics .  Two  basic  options  are 
presented  to  the  PRAG  member  for  descriptive  statistics  on  the  CPARS  data 
base.  The  PRAG  member  can  select  a  description  of  the  trends  for  Block 
14  and  15  items  over  initial,  intermediate,  and  final  assessments  for  a 
selected  contract;  or  the  PRAG  member  may  select  a  parent  corporation  or 
division/subsidiary  name  and  receive  descriptions  of  the  information 
across  many  contracts.  The  transaction  flow  diagram  for  this  activity  is 
p i c l  ured  in  F i gur  e  5-13. 

When  choosing'  the  second  option  the  PRAG  member  may  use  any  of  the 
relevancy  criteria  set  during  the  planning  process  to  limit  the  size  of 
the  data  base.  Reports  can  be  viewed  on  the  monitor  or  made  available 
for  printing. 

Find  Relevant  Informat  ion.  Using  the  selected  relevancy 
criteria,  the  PRAG  member  can  search  through  the  CPARS  data  base  for 
assessment  reports  which  are  pertinent  to  the  program  under  evaluation. 
The  search  capability  will  key  on  information  which  is  identified  iri 
standard  fields  and  on  information  which  may  or  may  not  be  provided 
within  certain  fields  on  the  assessment  report. 

The  PRAG  member  has  three  different  options  for  the  format  in  which 
in  format  ion  from  the  CPARS  data  base  will  be  presented.  The  PRAG  member 
can  choose  to  viev/print,  the  entire  assessment  report  as  the  Air  Force 
Systems  Command  Form  125,  create  a  unique  report  format,  or  select  one  of 
the  standard  report  forms  provided  by  the  system.  When  viewing  the 
report  on  the  monitor,  the  PRAG  member  will  be  able  to  move  both  backward 
and  forward  through  the  document  and  leave  the  report  from  any  location 
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PRAG  Member 


Provide  descript  ive  stat  ist.  ics . 


Figtire  5-13.  Provide  Descriptive  Statistics 
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within  the  document.  The  system  should  allow  the  PRAG  member  to  search 
and  sort  assessment  reports,  scan  descriptive  information  arid  color 
rating's,  and  focus  on  specific  comments  in  the  program  director/manager 
narrat ive. 

The  PRAG  member  also  has  the  option  to  extract  key  information 
elements  into  a  word  processing  file  where  they  can  be  summarized  and 
then  transferred  to  a  standard  or  unique  report  format.  The  transaction 
flow  dicigram  for  this  activity  is  depicted  in  Figure  5-14. 

Perform  Risk  Assessment .  Performance  Risk  Assessment  is  a 
two  step  process  which  first  checks  to  determine  each  programs’  relevance 
to  the  effort  being  considered  and  then  subjectively  summarizes  across 
all  contracts  so  that  a  risk  assessment  for  each  area  could  be  assigned. 
Although  not  specifically  requested  by  any  PRAG  member,  this  process 
seems  to  lend  itself  to  the  use  of  a  decision  support  system. 

The  PRAG  mtmber  would  first  consolidate  all  the  pertinent  comments 
and  rat  ings  from  the  relevant  assessment  reports.  Using  the  decision 
support  software,  the  PRAG  member  would  assign  the  relative  importance  of 
each  of  the  source  selection  evaluation  criteria  to  the  performance  risk 
assessment.  Depending  on  the  program,  this  could  be  at.  the  area  or  item 
level.  The  PR\G  member  then  looks  at  the  information  which  he  determined 
to  be  relevant.  Using  the  decision  support  software,  the  PRAG  member 
determines  the  relative  importance  of  each  example  of  contractor 
performance  to  the  evaluation  criteria. 

The  support  software  in  no  way  would  be  making  the  decision  for  the 
PRAG  member;  rather,  it  simply  provides  a  way  to  organize  the  various 
elements  of  information.  This  activity  is  depicted  in  Figure  5-15. 
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PRAG  Member 


Perform  risk  assessment . 


Figure  5-15.  Perform  Risk  Assessment 
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Generate  Reports  and  Prepare  Output .  The  transaction  flow 
diagram  shown  in  Figure  5-16  addresses  the  required  output  to  support 
PRAG  analysis  and  the  reports  needed  to  support  CPARS  administration. 

Output  will  be  available  both  for  display  on  the  computer  screen  and 
in  hard  copy.  The  system  will  provide  copies  of  assessment  reports, 
standard  format  summary  reports,  output  of  descriptive  statistics,  and 
user  defined  formats  to  support  the  PRAG  member's  tasks. 

The  CPARS  focal  point  has  three  basic  reports  \*hich  must  be  created 
to  support  the  administration  of  the  CPARS.  The  first  type  includes  the 
reports  submitted  quarterly  or  more  frequently  to  Air  Force  Systems 
Command  describing  the  latest  assessment  reports  entered  into  the  system. 
These  reports  must  state  the  contractor  name,  subsidiary  of  division, 
contract  number,  period  or  performance,  and  a  brief  description  of  the 
contract.  All  of  this  information  is  available  directly  from  the  CPARS 
data  base.  Also  included  in  the  first  type  of  report  is  the  requirement 
for  each  focal  point  to  submit  annually  a  report  which  identifies  the 
name  and  address  of  contractor  divisions  and/or  subsidiaries;  parent 
corporations;  the  number  of  times  each  contractor  has  submitted 
proposals;  and  any  new  offerors  for  inclusion  in  the  data  base. 

The  second  type  of  report  captures  information  from  the  tracking  log 
which  is  consistently  updated  during  the  data  collection  and  input 
processes.  The  report  will  be  used  for  identifying  slow  points  in  the 
system  and  to  provide  informal  ion  to  support  local  requirements. 

The  final  type  of  report  which  can  he  generated  by  the  CPARS  focal 
point  captures  information  from  the  contractor  name/address  data  base. 
Reports  are  sent  to  contractors  yearly  to  correct  the  information. 
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Figure  5-1(5.  Generate  Reports  and  Prepare  Output 


VI.  Conclusions  and  Recommendat ions 


This  chapter  describes  the  conclusions  reached  as  a  result  of  this 
research  effort  and  offers  recommendat ions  for  additional  steps  to  ensure 
the  successful  implementation  of  an  automated  system  which  can  support 
the  performance  risk  assessment  process. 

The  four  objectives  established  early  in  this  study  are  printed 
below.  The  objectives  proved  to  be  an  invaluable  tool  for  maintaining' 
the  focus  of  this  project  through  the  research  process. 

Objective  1.  Define  the  information  re<;ui  red  and  procedures  used  by 
the  Performance  Risk  Assessment  Group  (PRAG)  to  institute  the  current 
policy  for  providing  a  risk  assessment  to  support  the  source  selection 
det us i on  process ; 

Objective  2.  Determine  the  extent  to  w hich  information  derived  from 
a  fully  operational  Contractor  Performance  Assessment  Reporting  System 
(CPARS)  can  support  the  performance  risk  assessment  process; 

Objective  3.  Determine  the  administrative  procedures  used  to 
collect,  process,  distribute,  and  protect  contractor  performance 
informat  ion  under  the  CP.4RS;  and, 

Object  ive  4.  Establish  the  requirements  baseline  and  conceptual 
design  for  an  automated  information  system,  based  on  the  CPARS,  to 
collect,  process,  protect,  and  disseminate  contractor  performance 
assessments. 


Implementation  of  Object ives 

A  tailored  approach  to  the  traditional  method  for  software  systems 
development  was  chosen  to  design  art  automated  informat  ion  system  to 
assist  the  performance  risk  assessment  process.  This  research  effort 
implemented  the  first  three  steps  of  the  tailored  development 
methodology.  The  five  steps  which  comprised  the  methodology-  for 
information  systems  development  is  described  in  Figure  6-1. 
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Step  1.  Problem  Definition 


Step  3.  Development/Design 


Def ine  Scope 
Set  Objectives 

Step  2.  Requirements  Analysis 

Determine  Requirements 
Establish  Baseline 


-  Conceptual  Design 
Physical  Design 

Step  4.  Implementation 

Step  5.  Verification/Validation 


Figure  6-1.  Tailored  Approach  to  Systems  Development 

Implementation  of  the  requirements  analysis  step  resulted  in  a 
complete  description  of  the  procedures  used  and  the  information  required 
to  support  the  performance  risk  assessment  process.  The  role  of  the 
CPARS  as  a  primary  source  of  information  to  support  this  process  was 
described  and  assessed.  The  administrative  procedures  used  to  collect, 
process,  distribute,  and  protect  contractor  performance  information  were 
determined.  The  completion  of  these  steps  fulfilled  the  first  three 
objectives  of  the  research  process. 

The  information  collected  from  FRAG  members,  CPARS  focal  points,  and 
applicable  regulations  was  synthesized  into  a  clear  statement  of  the 
required  capabilities  for  the  automated  information  system  to  support 
performance  assessment.  The  requirements  baseline  provides  a  sound 
foundation  for  the  future  development  and  implementation  of  the  CPARS 
automated  information  system. 

The  research  process  followed  the  tailored  methodology  through  the 
development  of  a  conceptual  model  for  the  system.  The  conceptual  model 
developed  describes  the  applications  necessary  to  support  the  performance 
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assessment  process.  The  focus  of  the  conceptual  design  was  the  user 
of  the  system.  The  conceptual  model  demonstrates  how  the  applications  of 
the  information  system  can  assist  the  user  in  the  various  steps  of  the 
assessment  process. 

The  completion  of  the  requirements  baseline  and  the  conceptual 
design  for  the  automated  information  system  fulfilled  the  final  objective 
of  this  research  project. 

Although  the  physical  design  arid  actual  implementation  of 
applications  were  beyond  the  scope  of  this  research  effort,  sufficient 
time  allowed  for  the  development  of  an  initial  capability  which  builds  on 
and  enhances  the  tools  currently  in  use  at  Aeronautical  and  Ballistic 
Systems  Divisions. 

An  initial  capability  was  developed  using  the  integrated  software 
package  Enable  (Version  2.15)  and  an  IBM  XT  compatible  computer  system. 
Enable  was  used  to  create  a  data  base,  input  forms,  and  several  report 
forms  which  combine  the  basic  functions  of  the  automated  capabilities  at 
both  Ballistic  <'md  Aeronautical  Systems  Divisions.  Functions  were 
designed  to  allow  for  flexibility  and  growth,  considering  that 
implementation  may  be  dependent  on  the  eventual  capabilities  of  this 
system.  A  description  of  the  system  design  and  the  operating  procedures 
is  presented  in  Appendix  G.  A  copy  of  the  software  can  be  obtained  from 
the  Director  of  Research,  AFIT/LSC  Wr ight -Patterson  AFB  OH  45433-06583. 

Conclusions  and  Recommendat ions 

State  of  the  Performance  Assessment  Ini t iat ives.  The  results 
achieved  by  meeting  the  first  three  objectives  provide  an  account  of  the 
performance  risk  assessment  and  contractor  assessment  report ing  processes 
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as  they  have  been  implemented  across  the  five  product  divisions.  This 
assessment  of  the  current  state  of  the  system  comes  at  a  critical  point, 
approximately  one  year  after  the  processes  were  initiated.  Differences 
in  procedures,  methodology  and  results,  problems  with  current 
implementation,  lessons  learned,  and  suggested  changes  have  carefully 
been  reported.  The  performance  assessment  initiatives,  a  1  though  still 
relatively  new,  have  become  an  integral  part  of  the  source  selection 
process  tliroughout  Air  Force  Systems  Command. 

There  is  diversity  in  the  implementation  of  the  performance 
assessment  initiatives  among  the  five  product  divisions.  In  some  cases 
the  differences  are  driven  by  the  unique  requ i remen t s  and  cons iderat ions 
of  each  product  division.  However,  differences  can  also  be  attributed  to 
a  lack,  of  definitive  central  guidance.  Air  Force  System  Command 
supplements  to  Air  Force  Regulations  70-15  and  70-30  have  still  not  been 
updated.  These  documents,  when  final,  will  provide  official  guidance  for 
Air  Force  Systems  Command  source  selections  and  for  the  use  of 
Performance  Risk  Assessment  Groups  as  part  of  the  source  selection 
process . 

Recommendat i on .  Each  product  division  should  compare  and 
assess  the  varied  procedures  used  to  implement  the  performance  risk 
assessment  and  performance  assessment  reporting  processes.  The  best 
procedures  should  be  selected  to  ensure  an  accurate  assessment  of  a 
contractor’s  past  performance  is  provided  to  the  Source  Selection 
Author i ty . 

Deve 1 opment  of  an  Automated  Informat  ion  System.  This  research 
showed  that  a  baseline  of  system  level  requirements  could  in  fact  be 
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determined  which  would  accurately  describe  an  automated  system  based  on 
CPARS  to  support  the  performance  risk  assessment  process.  This  research 
also  demonstrated  that  the  requirements  identified  could  be 
conceptualized  into  a  design  for  the  automated  information  system. 

Recommendat i on .  The  capabilities  reported  in  the  requirements 
baseline  should  be  validated  and  prioritized  for  implementation. 
Standardization  of  procedures  and  methodologies  where  applicable  should 
be  sought  across  product  divisions  in  order  to  simplify  the  development 
process.  The  information  system  development  methodology  should  be 
carried  through  to  the  implementation  and  validation  of  an  automated 
capability  to  support  the  performance  assessment  process. 

Col lect ing  Pert inent  Informal  ion  to  Support  Performance  Ri  sk 
Assessment .  The  rigorous  process  which  an  assessment  report  must  go 
through  before  it  is  entered  into  the  system  is  a  key  factor  in  the  use 
of  the  information  to  support  source  selection  decisions.  The  contractor 
and  product  division  review,  along  with  the  practice  of  retaining  both 
preliminary  and  revised  assessments,  add  to  the  assessment  report’s 
credibi 1 ity. 

The  effort  expended  by  program  managers  when  developing  contractor 
performance  assessments  is  extremely  important  to  the  success  of  the 
CPARS  to  provide  pertinent  information  to  the  PRAG  members.  The  level  of 
detail  in  the  evaluations  and  the  overall  quality  of  comments  in  the 
narrative  sections  were  frequently  mentioned  by  both  PRAG  members  and  the 
CPARS  focal  points  as  areas  which  need  to  be  monitored. 

Recommendat i on .  Discussions  should  be  held  with  program 
managers  to  determine  their  personal  needs  when  executing  a  performance 
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assessment.  Identified  needs  should  be  incorporated  into  the 
requirements  baseline  and  conceptual  design.  Continued  use  of  training 
and  supporting  materials  are  necessary  to  ensure  adequate  attention  is 
focussed  on  the  program  manager’s  narrative.  Applications  for  the 
automated  system,  such  as  use  of  key  words  and  probing  questions,  should 
be  investigated  to  assist  the  program  manager  when  writing  a  performance 
assessment  report. 

Support  for  Automat  ion.  Development  of  an  automated  system  based  on 
CPARS  to  support  the  performance  risk  assessment  process  appears 
achievable.  It  is  reasonable  to  conclude  that  an  automated  information 
system  would  provide  benefit  to  both  the  PRAG  members  and  CPARS  focal 
points.  In  fact,  this  has  been  demonstrated  in  the  field  with  the 
existing  capabilities  at  Aeronautical  and  Ballistic  Systems  Division. 

Nearly  every  person  interviewed  indicated  that  the  CPARS  should  be 
automated.  The  conditions  appear  right  to  implement  the  automated  system 
as  long  as  the  question  of  how  assessments  can  be  easily  entered  into  the 
sys tern  is  answered . 

Recommendat i on .  .An  analysis  should  be  accomplished  to  make 
sure  that  the  benefits  associated  with  an  automated  information  system 
exceed  the  amount  of  work  which  will  be  expended  for  development, 
implementation,  and  maintenance  of  the  system.  A  specific  point  which 
must  he  looked  at  is  whether  the  time  required  to  enter  the  performance 
assessment  report  into  the  system  is  offset  by  time  savings  on  developing 
future  assessments  and  benefits  to  CPARS  administration  and  PRAG 
analysis.  This  activity  should  be  done  in  combination  with  a 
demons t rat  ion  of  a  prototype  syst  em.  A  microcomputer  based  system 


6-6 


similar  to  the  one  developed  using-  the  integrated  software  package  finable 
could  Lie  used  to  demonstrate  the  concepts  and  provide  actual  data  to 
support  the  analysis. 

Cons  i  do  rat  ion  of  the  Accfu  i  s  i  t  i  on  Management,  [nformat  ion  System 
for  a  Central ized  CPARS.  The  Acquisition  Management  Information  System 
(AMIS)  is  art  existing  network  of  computers  located  at  all  Air  Force 
System  Command  buying  activities  and  Air  Force  Plant  Representative 
Offices .  AMIS  provides  Air  Force  Systems  Command  with  an  immediate 
access  numagement  informat  ion  system,  an  automated  contracting  function, 
and  automated  administration  arid  payment  function  ( 8 ) . 

AMIS  is  right  in  the  middle  of  a  major  upgrade  to  enhance  user 
support,  increase  development  and  maintenance  productivity,  and  provide 
distributed  data  base  applications  (8).  This  is  being-  accomplished  with 
the  addition  of  a  state  of  the  art  relational  data  base  and  through  the 
use  of  a  4th  generation  language.  The  Integrated  Distributed  System 
upgrade  is  a  phased  approach  which  will  add  enhanced  capabilities  over 
the  next  several  years. 

The  Integrated  Distributed  System  upgrade  will  provide  additional 
capability  which  may  he  beneficial  to  contractor  performance  assessment. 
The  fact  that  AMIS  is  an  existing  network  which  ties  together  all  Air 
Force  Systems  Command  assets  is  a  positive  point  which  should  not  be 
overlooked.  It  is  possible  that  the  PRAG's  entire  information 
requirement  max  In'  satisfied  by  a  performance  assessment  automated 
information  system  which  connects  to  or  is  an  integral  part  of  the 
Acquisition  Management  Information  System. 
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According  to  Mr.  Bailey,  the  CPARS  automation  could  possibly  be  a 
"tag-along"  effort  t.o  Phase  IV  of  the  Integrated  Distributed  System 
upgrade  which  is  scheduled  for  implementat ion  sometime  in  the  middle  of 
1991  (8).  Mr.  Bailey  is  in  charge  of  the  Functional  Systems  Division 
within  the  Acquisition  Management  Information  System  Organization. 

Recommendat ion.  The  automation  concept  developed  during  this 
research  effort  should  be  formally  presented  to  the  .AMIS  Automation 
Working  Group.  The  features  of  AMIS  should  be  carefully  looked  at  to 
determine  if  there  is  a  logical  match  to  the  CPARS  automation 
requi rements .  If  a  significant  match  exists,  the  AMIS  organization 
should  be  directed  to  prepare  an  impact  statement  for  including  these 
requirements  into  the  Integrated  Distributed  System  upgrade  scheduled  for 
implementation  in  mid  1991. 

Closing  Remarks 

The  Contract  Performance  Assessment  Reporting  System  and  the 
Performance  Risk  Assessment.  Group  were  introduced  as  part  of  Air  Force 
System  Command’s  initiative  to  emphasize  commercial-1  ike  practices  for 
awarding  contracts  to  responsible  sources.  The  members  of  the  team  who 
developed  CPARS  had  a  complete  understanding  of  the  problems  with 
previous  attempts  at  performance  evaluation  and  tracking  systems,  and 
purposely  designed  the  system  to  counteract  some  of  the  past  problems. 

The  success  of  CPARS  will  depend  on  its  ability  to  provide  credible 
information  in  a  manner  which  is  not  cumbersome  to  the  Performance  Risk 
Assessment  Groups,  the  program  managers,  or  the  CPARS  administrators. 

The  results  of  this  research  effort  provide  a  foundation  for  an  automated 
information  system  which  may  help  achieve  these  objectives. 
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Appendix  B:  In  formation  Describing  PRAGs  arid  DRAG  Represent  at  ives 

The  information  presented  in  this  appendix  describes  the  PRAGs 
investigated,  and  the  background  and  specific  responsibilities  of  PRAG 
representatives  interviewed.  This  section  also  provides  insight  into  the 
differences  across  the  five  product  divisions. 

Tables  B-l  and  B-2  describe  the  distribution  by  product  division  of 
individuals  interviewed,  and  number  and  size  of  PRAGs.  The  latter  part 
of  this  section  describes  the  actual  responsibilities  of  individuals 
according  to  position  held  in  the  PRAG. 

In  order  to  preserve  the  confidentiality  of  source  selection 
sensitive  information,  individual  PRAG  representatives  are  not  attributed 
to  a  specific  PRAG,  source  selection,  or  program.  Also,  no  specific- 
program  or  source  selection  information  will  appear  in  this  text. 

Distribution  of  Represent  at ives .  Tab'c  B-l  depicts  the 
distribution,  by  product  division,  of  PRAG  representatives  who 
participated  in  the  research  process.  Three  of  the  16  PRAGs  which  had 
looked  at  CPARS  as  a  source  of  information  had  only  one  representative 
identified  to  participate  iri  this  research.  All  others  were  represented 
b\  both  a  senior  and  a  junior  representative. 

Sgze  of  PRAGs.  Table  B-2  presents  the  number  of  participants  of 
PRAGs  by  product  division.  Aeronautical  Systems  Division  had  by  far  the 
largest  number  of  members  per  PRAG.  Of  the  five  looked  at,  four  PRAGs  at 
Aeronautical  Systems  Division  had  more  than  seven  participants.  A  more 
detailed  representation  of  this  is  presented  in  Appendix  D,  Table  D-l 
PRAG  Attributes  ( 1  tomograph i c s ) . 
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Table  B-l 


Distribution  Of  PRAGs  and  PRAG  Representatives  By  Product  Division 


Number  of  PRAG 

Product  Division  Number  of  PRAGs  Representatives  Interviewed 


Space  Systems  2  3 
Munitions  Systems  4  8 
Electronic  Systems  2  3 
Ballistic  Systems  3  5 
Aeronautical  Systems  5  10 

16  29 


Table  B-2 

PRAG  Attributes  (Size) 


Average  Number 

Product  Division  of  PRAG  Members  Largest 


Smal lest 


Space  Systems  5 

Munitions  Systems  3.5 

Electronic  Systems  2.5 

Ballistic  Systems  3 

Aeronautical  Systems  7.8 

Across  All  Product  4.8 

Divisions 


7  3 

4  3 

3  2 

4  2 

10  4 

10  2 
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FRAG  Respons i b i I i t i es .  Responsibilities  of  individual 
representatives  differed  depending  upon  the  position  in  the  PRAG. 
Participants  in  PRAGs  could  be  characterized  by  one  of  three  positions: 
PRAG  Chairperson,  PRAG  Assistant,  or  other  PRAG  member.  The 
responsibilities  of  each  position  appeared  consistent  across  the  product 
divisions.  A  detailed  description  of  responsibilities  has  been  included 
in  Appendix  D,  Table  D-2  Respons ibi 1 i t ies  of  PRAG  Members  by  Position. 

The  PRAG  chairperson  is  generally  responsible  for  the  overall 
"orchestration"  (19)  of  the  risk  assessment  process.  The  chairperson  is 
usually  a  senior  ranking  individual  (Lieutenant  Colonel,  Colonel,  or 
civilian  equivalent)  who  operates  at  the  director  or  deputy  director 
level  in  either  the  System  Program  Office  or  Contracting  Office  (5;  13; 
19;  23;  25;  28;  36;  41;  49;  52;  56).  However,  there  were  exceptions  to 
this  (32;  34).  It  is  the  chairperson’s  responsibility  to  make  sure  the 
right  people  and  the  appropriate  resources  are  available  to  perform  the 
risk  assessment.  The  PRAG  chairperson  is  normally  the  individual  who 
briefs  the  Source  Selection  Authority. 

PRAG  assistants  are  members  with  additional  administrative 
respons ibi 1 i t i cz ,  The  primary  responsibility  of  the  PRAG  assistant  is  to 
make  sure  there  enough  information  is  collected  and  that  it  is 
effectively  presented  to  the  people  performing  the  assessment  (7;  It);  15; 
19;  24;  51).  At  Aeronautical  Systems  Division  these  individuals  are 
referred  to  as  the  "PRAG  Secretariat".  PRAG  assistants  are  normally  more 
junior  in  grade  (Lieutenant,  Captain,  or  civilian  equivalent)  than  the 
chairperson,  and  other  PRAG  members  (6;  7;  18;  15;  24;  33;  37;  51). 
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The  assistant’s  responsibilities  include  both  administrative  and 
evaluation  functions.  Assistants  are  responsible  for  performing-  al 1 
correspondence,  sending  out  questionnaires,  conducting  interviews, 
scheduling  meetings,  and  acting  as  a  full  participating  member  in  the 
risk  assessment  (6;  7;  10;  15;  24;  33;  37).  Of  all  PRAG  representatives 
interviewed,  there  was  only  one  case  in  which  the  assistant  was  not 
involved  in  assessment  of  information  (51). 

In  general,  PRAG  members  are  responsible  for  evaluating  all  sources 
of  available  information,  and  providing  a  performance  risk  assessment. 
With  the  exception  of  Aeronautical  Systems  Division,  grade  of  members 
varied  widely.  At  Aeronautical  Systems  Division  PRAG  members  were 
typically  director  level  personnel,  while  at  the  other  product  divisions 
PRAGs  would  comprise  a  mix  of  junior,  mid-range,  and  director  level 
personnel.  PRAG  members  bring  experience  and  knowledge  from  their 
functional  backgrounds  to  the  group. 

At  Ballistic,  Electronic,  Space,  and  Munitions  Systems  Divisions, 
each  member  would  likely  participate  in  the  actual  gathering  of  the 
information.  This  was  not  the  case  for  four  of  the  five  PRAGs  at 
Aeronautical  Systems  Division  where  information  was  collected,  collated, 
and  organized  by  the  PRAG  Secretariat  and  then  presented  to  the  PRAG 
members  (7;  10;  13;  15;  19;  23;  23;  44). 

Add i t. i ona  1  Source  Selection  Responsibi  1  ities.  Nearly  all 
representat i ves  interviewed  at  Aeronautical  Systems  Division  (8  of  10) 
and  all  at  Electronic  Systems  Division  (3  of  3)  had  responsibilities 
dedicated  to  participation  in  the  PRAG  only.  This  was  not  the  case  at 
the  other  product  divisions.  Often  PRAG  representatives  would  have 
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source  selection  duties  in  addition  to  their  PRAG  responsibilities.  At 
Space  Systems  Division  all  of  the  PRAG  representatives  had  additional 
source  selection  responsibilities.  Half  of  the  PRAG  representatives 
from  Munitions  Systems  Division  (4  of  8  interviewed)  and  just  less  than 
half  of  PRAG  representatives  interviewed  from  Ballistic  Systems  Division 
(2  of  5)  had  additional  source  selection  duties.  Overall,  11  of  the  29 
PRAG  representatives  had  source  selection  responsibilities  in  addition 
to  the  PRAG .  A  detailed  description  of  additional  source  selection 
responsibilities  by  PRAG  has  been  included  in  Appendix  D,  Table  D-3 
Additional  Responsibilities  Matrix. 


Appendix  C:  Survey  Instrument s 


PRAG  Members  Questionnaire 

Your  name  was  supplied  by  your  Product  Division  Source  Selection 
Office  as  a  participant  in  a  Performance  Risk  Assessment  Group.  The 
purpose  of  this  project  is  to  find  out  what  types  of  information  are 
required  by  the  PRAG  and  to  see  if  an  automated  version  of  the  Contractor 
Performance  Assessment  Reporting  System  can  help  meet  these  needs. 

I’d  like  to  say  right  up  front,  that  I’m  aware  of  the  sensitivity  of 
source  selection  material,  and  I’m  not  interested  in  specific  program 
informal  ion.  What  I  really  need  t,o  find  out  are  the  general  procedures 
assoc iat ed  with  the  performance  risk  assessment  process.  To  do  this  I  am 
interviewing  knowledgeable  persons  like  yourself  who  have  experienced  a 
PRAG  f i rs  thand . 

1.  For  the  PRAG  which  you  participated  in,  what,  were  your  specific 
responsibilities'?  Did  your  position  have  a  title?  (What  was  it?)  How- 
many  other  participants  were  there?  How  did  their  responsibilities 
differ  from  yours?  Did  you  have  any  other  source  selection 

resnons ib i 1 i t i es  bes ide  the  PRAG? 

2.  Could  you  step  me  through  the  order  of  events  which  took  place  from 
the  time  you  were  notified  of  your  participation  --  to  the  briefing  of 
the  results  to  the  source  selection  authority'7  Could  you  describe  what 
happened  at  this  first  meeting?  Were  there  any  organizational 
instructions?  How  often  did  you  meet  and  for  how  long?  Were  the 
responsibilities  divided  among  you? 

3.  I’m  particularly  interested  in  whether  there  was  any  compiiter  support 
to  assist  the  group  during  this  process.  (That  is,  did  you  use  computers 
for  any  of  the  following:  word  processing,  graphics,  spread  sheet  or 
statistical  analysis,  or  data  base  manipulation?) 

4.  I’d  like  now'  to  pin  down  the  kinds  or  categories  of  information  which 
were  required  in  order  for  the  group  to  provide  a  performance  risk 
assessment .  What  kinds  of  contractor  performance  information  were 
provided  to  the  group  to  make  the  risk  assessment?  What  were  the  sources 
for  the  information?  What  kinds  of  information,  other  than  performance 
related,  were  provided?  Where  did  this  information  come  from? 

Would  you  describe  each  of  the  categories  of  infor*’--’*  :on  categories 
as  being  common  to  all  programs,  or  were  they  specific  to  this  particular 
effort? 

Assuming  that  any  constraints  provided  by  current  regulations  are 
not  changed,  what  other  types  of  information  could  have  assisted  the  risk 
assessment  process  but  were  not  presented  to  the  PRAG?  (In  other  words, 
was  there  any  information  missing,  that  is  not  restricted  by  current 
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regulations,  which  you  would  have  liked  to  have?)  Where  would  you  go  to 
find  this  information?) 

5.  The  information  gathered  from  the  various  sources  you  identified 
earlier  had  to  be  compiled  and  organized  in  some  way  in  order  for  it  to 
be  useful  to  the  PRAG  members.  Obviously  someone  put  a  lot  of  work  to 
put  this  folder  together  for  the  group,  did  you  participate  in  this 
process? 

Yes  No 

If  Yes: 

Starting  with  each  of  the  sources  you  identified  earlier,  could  you 
describe  the  step  by  step  process  in  which  the  data  from  the  various 
sources  were  compiled  and  organized  to  facilitate  the  risk  assessment. 

Was  information  from  more  than  one  source  ever  combined?  Was  there 
adequate  time  available  to  organize  the  data  collected?  How  much  time 
did  this  entire  process  take?  Did  you  have  any  difficulties  associated 
with  the  compiling  and  organizing  process?  (Could  you  please  describe 
them?)  Were  there  any  types  of  information  that  would  have  been  useful 
to  the  PRAG,  but  were  not.  included  in  the  folder  presented  to  them 
because  the  processing  of  the  available  raw  data  was  too  difficult  or  you 
simply  did  not  have  the  means?  What  were  they? 

If  No: 

As  the  recipient,  it  is  critical  that  the  information  be  presented 
to  you  in  a  useful  manner.  Do  you  know  who  prepared  the  information  used 
by  the  PRAG?  (Who?)  Are  you  aware  of  any  difficulties  they  may  have  had 
with  the  compiling  and  organizing  process?  (Could  you  please  describe 
them?)  How  useful  was  the  information,  in  the  format  which  it  was 
presented? 

6.  Could  you  please  describe  the  kinds  of  physical  evidence  (charts, 
graphs,  tables,  etc.)  you  had  in  front  of  you  while  making  the  risk 
assessment?  Could  you  please  describe  the  types  of  output  used  by  the 
PRAG  brief  the  final  risk  assessment  to  the  Source  Selection  Authority? 

7.  Since  it’s  only  been  a  short  while  since  the  CPARS  was  officially 
established,  I’m  interested  to  know  which  of  the  following  phrases 
describes  how  much  you  know  about  the  Contract  Performance  Assessment 
Report ing  System? 

Nothing  A  little  bit  Quite  a  hit  A  lot 

For  the  PRAG  which  you  participated,  was  the  CPARS  looked  at  as  a 
potential  source  for  information  supporting  the  performance  risk 
assessment  ? 


If  YES,  then,  approximately  what  percentage  of  the  PRAG's  total 
informat  ion  requirements  were  supported  by  data  that  was  in  the  CPARS? 

Tf  you  could,  look  at  the  AFSC  Form  125  now,  to  help  jog  your 
memory,  and  tell  me  if  there  are  any  specific  fields  of  the  CPAR  which 
you  think  would  not  be  very  useful  as  a  source  for  information  to  support 
the  PRAG’s  tasks?  (What  are  they?) 

The  first  few  people  I  talked  to  indicated  that  there  was  some 
difference  in  the  frankness  of  the  comments  between  the  questionnaires 
and  the  CPARs  for  the  same  contract.  I  was  curious  if  you  had  the  same 
perception  and  if  you  could  possibly  expand  on  this? 

8.  Referring-  back  to  the  first  half  of  the  interview,  you  identified  the 
following  different  types  of  information  needed  to  support  the 

performance  risk  assessment  process .  Can  these  types  of 

information  be  supported  from  data  in  the  CPARS?  Comparing  this  to  the 
AFSC  Form  125,  could  you  please  identify  the  specific  fields  of  the  CPAR 
which  could  support  this  type  of  information  needed?  Can  the  data  in 
these  fields  be  used  directly  in  the  CPARS  format,  or  must  they  be 
processed  in  some  way?  (Can  you  identify  the  processing  steps  necessary 
to  transform  each  of  the  data  elements  to  the  information  required?)  How 
is  the  narrative  section  used  to  augment  the  ratings  supplied  on  the 
front  of  the  form? 

9.  Based  on  your  experience  with  this  process  are  there  any  things 
which  you  think  a  designer  for  an  automated  informat ior  system  based  on 
the  CPARS  should  concentrate  on?  How  about  things  to  avoid? 

10.  How  would  you  expect  an  automated  information  system  to  work  for  you 
in  providing  a  performance  risk  assessment '?  What  capabilities  should  it 
have?  What  functions  should  it  perform?  Can  you  describe  an  example  of 
how  it  would  be  used? 

11.  If  an  automated  information  system  were  developed  which  could 
provide  information  to  assist  PRAGs  in  their  risk  assessment.,  who  do  you 
think  would  be  appropriate  to  sit  in  front  of  a  computer  terminal  and 
generate  the  product?  Would  it  be  a  PRAG  member,  CPARS  focal  point,  or 
someone  else?  Please  explain. 

12.  After  experiencing  this  process  all  over  again,  can  you  think  of  any 
changes  which  could  be  made,  within  the  constraints  of  the  current 
regulations,  that  may  have  made  the  risk  assessment  process  more 
effective  or  more  efficient?  What  I’m  really  looking'  for  are  lessons 

1  earned . 
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Contractor  Performance  Assessment  Reporting  System 
Focal  Point  Questionnaire 


Your  answers  to  the  following  questions  will  help  me  determine  what 
administrative  procedures  you  use  to  collect,  process  and  distribute 
contractor  performance  information;  and,  how  these  procedures  would 
affect  the  architecture  for  an  automated  performance  assessment  reporting 
system. 

Respondent  Information: 

Full  Name: _ .  Date: _ / _ / _ . 

Duty  Title: _ . 

Address: _ _ _ . 


CPARS  Related  Responsibilities: 


1.  Please  describe  how  the  administrat ion  of  the  CPARS  is  organized  at 
your  product  division?  Is  there  one  central  office  for  CPARS 
administrat ion?  Is  there  a  CPARS  representative  located  in  each  system 
program  office?  What  are  the  responsibilities  of  each  of  the  different 
elements? 

2.  During  our  conversation  I  would  like  to  talk  about  the  contractor 
performance  assessment  process  as  it.  is  currently  implemented.  I'm 
interested  in  the  detailed  steps  and  the  peculiarities  of  the  process 
which  would  not  be  found  in  the  CPAR  regulation.  To  facilitate  the 
discussion,  I’ve  categorized  the  process  into  five  different  phases:  data 
collection,  data  input,  maintenance,  output,  and,  reporting. 

a.  Data  collection:  Using  the  guide  below  please  describe  how  the 
actual  assessment  report  is  developed.  Include  any  local  policies  or 
practices  which  are  distinctive  to  your  product  division. 

(1)  Process  initiation:  Where  is  the  process  initiated?  That 
is,  who  notifies  the  program  manager  to  begin  a  CPAR?  Is  it  the  system 
program  office,  source  selection  office,  CP.AR  focal  point,  or  somewhere 
else? 


(2)  Program  manager’s  evaluation:  Are  there  local  supplements 
to  the  CPAR  regulation  or  other  guidance  available  to  assist  the  program 
manager'.’  Is  there  a  continuing  training  program  for  program  managers? 
Where  do  the  program  managers  go  for  information?  Does  "Program 

Di rector /Manager"  mean  the  overall  Program  Director  or  is  it  the 
program/project  manager  for  the  individual  contract? 

(3)  Contractor’s  comments:  Who  actually  sends  the  CPAR  to  the 
contractor?  How  is  the  information  controlled  during  the  process?  Who 
does  the  contractor  send  the  CPAR  back  to? 
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(4)  Program  manager  update:  If  the  progTam  manager  decides  to 
update  the  initial  assessment,  this  is  done  on  a  new  (PAR  and  attached  to 
the  original.  Ts  the  entire  form  retyped,  or  just  the  modifications?  Is 
the  new  CPAR  sent  back  to  the  contractor  for  additional  comments?  If  so, 
is  there  a  limit  to  the  number  of  iterations? 

(5)  Review  Process:  Is  there  one  or  more  than  one  reviewing 
official  for  the  product  division?  Where  are  they  typically  located  in 
the  organization?  Does  the  reviewing  official  look  at  the  CPAR  prior  to 
its  being  sent  to  the  contractor?  What  happens  if  the  reviewing  official 
disagTees  with  the  program  manager’s  assessment  --  does  the  process  start 
over  again? 

b.  Data  input. :  Describe  how  and  how  often  each  CP.AR  is  entered 
into  the  command-wide  and  local  delta  liases.  Are  CP.ARs  filed  according  to 
parent  company,  division/profit  center,  other,  or  some  combination  of  the 
prior  categories?  Is  the  Department  of  Defense  Activity  Address 
Directory  Code  (DODAAD)  used  in  this  process? 

e.  Maintenance :  How  are  CP.ARs  updated  or  removed?  If  a 
corporation  is  merged  into  another,  how  would  the  data  base  be  updated? 
How  are  contractor  name  changes  handled?  What  would  happen  when  an 
assessment  becomes  obsolete  after  five  years?  The  regulation  allows  a 
program  manager  to  determine  when  a  CP.AR  should  kept  on  a  contractor 
for  longer  than  five  years.  How  is  tnis  implemented?  .Are  there  other 
maintenance  actions  which  must  be  performed  on  the  data  base? 

d.  Output  ( informa’ ion  product  to  support  the  PRAG) :  When  a  PRAG 
member  requests  information  from  the  CPARS,  what,  are  they  typically 
looking  for?  Do  they  request  CPARs  for  a  corporation  or  for  a  specific 
division  or  profit  center?  Do  they  ever  request  you  to  sort  through  for 
a  specific  type  of  contract  or  program?  Does  the  PRAG  representative 
have  access  to  the  original  files  or  are  copies  made  by  the  focal  point? 

e.  Report ing:  What  reports  do  you  need  to  generate  as  the  CPARS 
focal  point?  What,  do  they  look  like?  How  are  they  put  together?  How 
often? 

3.  Have  you  found  any  problems  inherent  to  this  process?  If  yes,  what 
are  they,  and  do  you  have  any  suggestions  for  recommended  improvements? 

4.  According  to  the  CPAR  regulation,  all  information  contained  in  the 
system  is  considered  Source  Selection  Sensitive.  What  special 
constraints,  for  the  transportation,  protection,  and  storage  of  this  data 
exist0  .Are  there  any  regulations  prohibiting  transfer  of  this 
information  over  electronic  media? 

f>.  /Are  there  ever  any  attachments  other  than  the  continuation  of  the 
narrative  which  will  need  to  remain  with  the  form?  If  so,  what  are  they? 
What,  are  they  used  for? 
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6.  How  many  reports  currently  exist  in  the  CHARS'?  How  large,  in  terms 
of  number  of  reports,  do  you  anticipate  the  data  base  to  get? 

7.  There  are  approximately  HO  contractors  being  reported  on  with  the 
CPARS.  Do  you  expect  this  number  to  increase  s igni f icant ly?  By  how 
much?  How  many  division/profit  centers  does  this  mean? 

8.  Do  you  know  of  any  on-line  data  source  which  has  a  complete  listing 
of  contractor-divis ion  addresses  and  identifies  the  parent  company?  Can 
this  be  tapped  into  to  get  the  information? 

3.  What  type  of  computer  system  do  you  have  in  the  office?  What 
software  packages  are  on  the  system? 

It).  The  five  phases  of  the  CHAR  process  which  you  have  described  in 
detail  are  printed  again  below.  I’d  like  to  ask  you  to  step  back  and 
think  about  each  phase  for  a  moment.  How  would  you  expect  an  automated 
information  system  to  work  for  you  in  each  phase  of  this  process?  What 
capabilities  should  it  have?  What  functions  should  it  perform? 


a.  Data  collection: 

(1)  Process  initiation: 

(2)  Program  manager ’ s  evaluation: 

(3)  Contractor’s  comments: 

(1)  Program  manager  update: 

(  5 )  Rev i ew  Process : 

b.  Data  input  : 

c.  Maintenance: 

d.  fXitput:  • 

e.  Report  ing: 


11.  In  talking  with  several  PRAG  members,  they  have  suggested  various 
capabilities  which  an  automated  version  of  CPARS  might  have  to  support 
their  informat  ion  needs.  <Xie  of  these  is  the  ability  to  search  the  CPARS 
to  identify  relevant  assessment  reports. 

If  an  automated  informat  ion  system  were  developed  which  could  help 
collect,  process,  ;ind  disseminate  contractor  performance  information  to 
assist  PRAGs  in  their  performance  risk  assessment,  who  do  you  think  would 
be  the  appropriate  person  to  sit  in  front  of  the  computer  terminal  arid 
generate  the  product?  Would  it  oe  a  PRAG  member,  CPARS  focal  point,  or 
someone  else?  Please  explain. 
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Append i x  D :  De t a i  led  Data  Prom  PRAG  Int erv ievs 


Table  D-l 

PRAG  Attributes  (Demographics) 


Product  Division 

Number 

Of  PRAG 
Members 

Grade  Range 

Of 

Personnel 

Functions  Represented 

Space  Systems 

3 

Mid-range 

Program  Manager  and 

Con  t  rac  t i ng 

Space  Systems 

7 

Di rector- ieve 1 , 
Mid-range  and 
Junior 

Program  Manager, 

Con t r ac t i ng ,  Test  anc i 
Ergi  neer  ing 

Nfun  i  t  i  ons  Sy s  t  ems 

•  t 

Mid-range 

Program  Manager, 

Contract ing  and 
Engineering 

Muii  i  1 1  ons  Sys  t  ems 

4 

Director- level 
and  Mid-range 

Program  Manager, 

Eng  i  nee  r ,  Con  t  rac  t,  i  ng 
and  Test 

Munitions  Systems 

4 

Mid-range 

Program  Manager  and 
Engineer ing 

Mun i t i ons  Sys  t ems 

3 

Mid- range 

Program  Manager,  Buyer 
and  Cent  rac  ting  Of  f i cer 

Electronic  Systems 

2 

Deputy  Director 

Support.  Systems  and 

Cont  ac  t.  ing 

E 1  ee  t  ron i c  Sys t  ems 

3 

Director- level 
and  Junior 

Deputy  Program  Direr' tor 
and  Program  Control 

Bal 1 ist ic  Systems 

4 

Di rector- level , 
Mid-range,  and 
Jun i or 

Program  Manager  and 

Cont  ract.  ing, 

Ba 1  list  ic  Sysf  ems 

3 

Di  rector- 1  eve  1  , 
Mid-range,  and 

Project  Officer  and 
Program  Control  Junior 

Ba : 1 ist  i c  Systems 

2 

Mid-range 

Project  Officer 
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Table  D-l  (Continued) 


PRAG  Attributes  ( Demographics ) 


Produc  t  Division 

Number 

Of  PRAG 
Members 

Grade  Range 

Of 

Personne 1 

Functions  Represented 

Aeronaut i ca 1  Sys  t  ems 

7 

Director- leve 1 
and  Mid-range 

Represent  all  functional 
areas,  ( Engineering, 
Program  Control, 

Cont  ract ing, 

Manufactur ing,  Test, 

Log  i  s  t  i  c  s  ,  Program 
Manager ) 

Aeronautical  Systems 

9 

Director  Level 
and  Junior 

Represent  all  Systems 
f line  t.  i  ona  1  areas 

Aeronautical  Systems 

1 

Director  Level 
and  Junior 

Technical,  Logistics 
Manufacturing  and 
Program  Management 

Aeronaut i ca 1  Sys  terns 

9 

Director  level 
and  Mid-range 

Represent  all 
f  une  t i ona 1  areas 

Aeronaut i ca 1  Sys  t  ems 

It) 

Director  Level 

Represent  all  functional 
areas,  ( Engineering, 

Test /Con f igurat ion, 
Logistics,  Contracting, 
Manufacturing,  Tactical 
Product,  Strategic  and 
Airlift) 
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Chairperson 


Space 
Sys  t  ems 


Mi  in  i  t  i  ons 
Systems 


Electronic 
Sys  terns 


Ballistic 

Systems 


Aeronaut ical 
Sys  t  ems 


Responsibilities  of  PRAG  Members  by  Position 


Responsibi 1  it ies 


Ensure  that  the  group  was  able  to  research  as  many  sources 
of  information  on  past  performance  as  practical  and 
available,  and  develop  an  analysis  for  the  benefit  of  the 
Source  Selection  Authority  an  assessment  of  the  past 
performance  (52). 

Report  to  the  Source  Selection  Evaluation  Team  and  Source 
Selection  Authority  what  the  performance  of  the  contractor 
had  been  on  previous  contracts.  Organize  the  team  seeing' 
that  the  process  is  executed  properly  and  prepare  final 
report  ( 28 ;  49 ) . 

Report  to  the  Source  Selection  Evaluation  Team  or  Source 
Selection  Advisory  Council  as  regulations  prescribe.  Make 
sure  everything  gets  done.  Act  as  communicator  with  the 
source  selection  org'anizat  i  on .  When  all  information  is 
received,  determine  what  information  is  relevant,  and 
what  comments  should  go  into  the  brief  and  vhat  should 
not.  Presented  briefing  to  Source  Selection  Evaluation 
Team/Advisory  Council  and  Source  Selection  Authority  (5). 

Take  an  active  role  in  the  assessment  process,  hiisure 
that  the  sources  of  information  are  available,  and  provide 
the  necessary  feedback  (36).  On  one  program  the  Chairman 
assisted  in  the  assessment,  but  performed  mostly  in  an 
advisory  capacity  (37). 

Organize  the  group  which  was  going  to  do  the  job,  define 
the  process,  execute  the  process,  perform  final  briefing 
to  Source  Selection  Authority  (23;  56).  Ensure  that 
appropr i ate  expertise  is  pulled  together  to  thoroughly 
evaluate  the  offerors  past  performance.  Make  sure  the 
entire  process  is  orchestrated,  that  the  questionnaires 
are  sent  to  the  appropriate  people.  Select  a  Secretariat 
to  perform  administrative  functions  (13;  19). 
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Table  D-2  (Continued) 


.Ass  istant. 


Space 

Systems 

Ballistic 
Sys  t  ems 


Aeronaut  ica 
Sys t  ems 


lies  pons  i  b  i  1  i  t  i  es  of  PRAG  Members  by  Pos  i  t  i  on 


Responsibi 1 ities 


In  charge  of  getting  the  contractor  proposal  appraisal 
reports,  summarizing  them  and  putting  everything  together. 
(33). 

Primary  responsibility  vas  to  make  sure  there  was  enough 
information  collected  to  perform  the  evaluation,  evaluate 
the  information,  perform  the  assessment,  and  brief  the 
Source  Selection  Authority  (37).  Collect  data,  collate 
data,  performed  analysis,  and  compile  briefing  (6). 

1  Secretariat  performs  administrative  tasks  so  that  the 
information  is  effectively  presented  to  the  people 
performing  the  assessment  (15;  21;  19).  Responsible  for 
getting  CPAR  information  and  distributing  information  to 
other  panel  members.  Develop  questionnaire  to  reflect 
areas  and  items  of  the  proposal  evaluation  criteria.  Send 
questionnaire  out  to  other  government  organizat ions . 
Consolidate  responses  <md  distribute  them  to  the  other 
panel  members  (10),  plus  act  as  a  full  participating 
member  in  the  risk  assessment  (7;  24).  The  Secretariat 
becomes  the  most  knowledgeable  person  for  that  PRAG .  Acts 
as  a  focal  point  for  all  questions  by  the  other  PRAG 
members  (15).  Helped  things  get  done  on  time. 
Administration  of  procedures.  Was  not  involved  in 
assessment  of  information  (51). 
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Members 


Space 

Systems 


Mun i t  i ons 
Sys t ems 


Electronic 

Systems 


Ba  1  1  i  s  t  i  c 
SVf; !  t'jjis 

Aeronaut i ca 1 
Syst ems 


Responsibilities  of  PRAG  Members  by  Position 


Respons i b i I i t i es 


Evaluated  CPARS  on  Contractor,  conducted  verbal 
ques t i onna i re  over  the  phone  (42).  Once  all  the  material 
was  gathered  together,  each  person  reviewed  it  and  came  up 
with  their  own  assessment.,  and  then  the  group  met  and  came 
up  with  a  general  consensus  (33). 

Provide  experience  from  the  contracting  field.  All 
members  were  assigned  a  number  of  .surveys  to  send  out  and 
interviews  to  conduct.  Conducted  interviews  with  program 
managers ,  procurement  contracting  officers  arid 
administrative  contacting  officers  and  assessed  risk. 
Responsibilities  were  split  among  the  members  (41).  Look 
at  other  sources  of  information  and  evaluate  the 
contractors  past  performance  (57).  Need  at  least  3  people 
on  the  team  in  order  to  break  deadlocks  (29).  Members  had 
very  little  interfa<  c  with  the  Source  Selection  Evaluation 
Team  and  Source  Selection  Authority,  but  all  three  members 
shared  responsibility  for  assessment  (49). 

Evaluate  each  offerors  past  performance,  concentrating  on 
functional  expertise  (38;  45).  Considerable  amount  of 
time  making  charts,  keeping  records,  contacting'  people  and 
sending  out  questionnaires  (38). 

Reviewed  questionnaires  and  CPARs ,  assess  risk,  and 
prepare  briefing  (12). 

Participated  on  PRAGs  under  both  individual  assessment  and 
team  assessment  approaches.  Responsible  to  go  through  all 
data  and  assign  risk  from  own  prospective  to  each  item 
assigned  for  all  the  offerors.  Responsible  for  reviewing 
all  information  and  assign  risk  to  each  item  assigned  for 
all  offerors.  Each  member  also  must  be  prepared  to  supply 
support  each  assessment.  Under  teaming  approach  members 
were  assigned  along  the  lines  of  their  functional 
expertise  (44).  Members  bring  specific  expertise  and 
knowledge  to  the  group  ;tnd  are  responsible  to  review 
contractor  provided  information  and  government  provide:! 
information  and  perform  analysis  based  on  their  area  of 
expert i se  ( 7 ;  10;  13;  1 5 ;  23 ;  24 ) . 


Table  D-3 


Additional  Responsibilities  Matrix 


Product  Division 

PRAG  Position 

Additional  Source  Selection 
Responsibil ities 

Space  Systems 

PRAG  Chairperson 

Contracting  Officer,  Head  of 
Cost  Team 

Space  Systems 

PRAG  Member 

Also  responsible  for  source 
selection  evaluation  at  the 
factor  level  and  was  an  item 
chief 

Space  Systems 

PRAG  Member 

Also  a  member  of  the  technical 
and  managemen  t  evaluation 
teams 

Mun  i  t.  i  ons  Sys  t  ems 

PRAG  Member 

Program  Manager 

Mun i t  i on  Sys  t  ems 

PRAG  Chairperson 

None 

Mun i t i on  Sys terns 

PRAG  member 

Procurement  Contracting  Officer 
for  the  contract 

Mun i t i on  Sys  t  ems 

PRAG  Chairperson 

None 

Mun i t ion  Systems 

PRAG  Member 

None 

Munition  Systems 

PRAG  Chairperson 

None 

Ntun  i  t  i  on  Sy s  t  ems 

PRAG  Chairperson 

Source  Selection  Eval nation 
Conmittee  Chairperson 

Mun i t  i on  Sys  terns 

PRAG  Member 

Chairperson  Cost  Committee  and 
Proc  ur emen  t  Con  t  rac  t i ng 

Of  f  i  cer 

Electronic  Systems 

PRAG  Chairperson 

None 

F,  1  ec  t  ron  i  c  Sys  terns 

Advisor  to  PRAG 

None 

Fleet rnnie  Systems 

PRAG  member 

None 

Ballistic  Systems 

PRAG  Chairperson 

None 

Ballistic  Sys  t  ems 

Assistant  to 

PRAG  Chairperson 

None 

P-(> 
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Additional  Responsibilities  Matrix 

Add i t i ona 1  Source  Selection 


Product  Division 

PRAG  Position 

Responsibi 1 ities 

Ballistic  Systems 

PRAG  Chairperson 

Source  Selection  Evaluation 
Team  Chairperson 

Ballistic  Systems 

PRAG  Member 

None 

Ballistic  Systems 

PRAG  Vice 
Chairperson 

Source  Selection  Evaluation 
Team  Vice  Chairperson 

Aeronaut ical 

Sys  t  ems 

PRAG  member 

None 

Aeronaut ical 

Systems 

PRAG  Chairperson 

None 

Aeronaut ical 

Systems 

PRAG  Secretariat 

None 

Aeronaut,  ical 

Systems 

PRAG  Chairperson 

Source  Selection  Evaluation 
Board  Chairperson 

Aeronaut ical 

Systems 

PRAG  Assistant 

Source  Selection  Executive 
Of  f icer 

Aeronaut ical 

Systems 

PRAG  Chairperson 

None 

Aeronaut ical 

Sys terns 

PRAG  Secretariat 

None 

Aeronaut ical 

Syst  ems 

PRAG  Chairperson 

None 

Aeronaut ical 

Systems 

PRAG  Secretariat 

None 

Aeronaut ical 

Sys  t  ems 

PRAG  Member 

None 
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Past  Use  of  Computers  to  Support  PRAGs 


Space  Systems 


Mun  i  t  i  orts  Sy s  t  ems 


F 1 ec t ron i c  Systems 


Ballistic  Systems 


Aeronautical  Systems 


Used  for  word  processing'  and  chart  making; 
mostly  used  for  the  preparation  of  the 
briefings.  (33;  42;  52) 

Availability  on  the  computers  was  tight.  Had  to 
get  there  on  off  hours  because  there  was  always 
someone  using  them.  (42) 

Word  process  i rig  used  to  type  the  ques t  i onna i re , 
otherwise  none.  (16;  29) 

Computer  support  used  for  word  processing  and 
chart  making  (28;  32;  49;  57) 

Computers  there,  but  were  not  used.  (41) 


Used  computers  for  word  processor  and  graphics 
(5;  38) 

No  computer  support  at  all,  CPAR-quest i onna ires 
typed  up  on  electric  typewriters  (45) 

Computer  support  limited  to  word  processing  and 
chart  building.  (6;  25;  36) 

Word  Perfect  was  used  for  the  questionnaire 
tracking  data  base,  word  processing  and  slide 
preparation.  (12) 

Computers  used  for  word  processing  and  graphics 
(Harvard  Graphics).  Templates  were  made  up  for 
charts  and  some  of  the  sections  of  RFP  and 
reports.  Sending  copy  of  floppy  disk.  (37) 

Secretariat  used  Z-248  for  word  processing  and 
briefing  development  (7;  10;  19;  23;  23;  33;  51; 
56) 


Used  spreadsheets  for  own  work  (PeaehCalc). 
Used  contractor  lists  for  many  different 
purposes.  Data  received  verses  contractor 
supplied  data  verses  relevant  data.  Need  for 
notebooks  for  the  members.  Administrative 
assistants  used  Harvard  Graphics  for  Final 
Briefing.  (15) 


Table  D-5 


PRAG  Representatives’  Lessons  Learned 

1.  Dedicated  PRAG  teams,  separate  from  source  selection  evaluation.  (42) 

2.  To  add  credibility  need  more  specifics  on  the  performance.  Take  more 
care  to  document.  Need  to  expand  the  CPARS  data  base.  The  SPO  which 
this  individual  works  looks  at  many  contractors  which  are  not  on  the 
list.  Would  like  to  have  a  local  system  for  these  small  contractors, 
but  there  is  resistance  to  do  that.  In  process  of  trying  to  add 
contractors  to  overall  system.  They  have  found  that  CPARS  is  great 
for  providing  past  performance  information,  but  also  the  report  card 
aspect  of  it  is  really  get  the  contractors  attention.  It  is  a 
helpful  tool  to  motivate  the  contractor.  (49) 

3.  Would  have  been  nice  to  have  description  of  process  up  front.  Also 
be  more  organized  so  that  each  member  performs  the  fund  ions  in  same 
ways.  Responsibilities  for  the  PRAG  should  be  better  defined.  On 
the  makeup  of  the  PRAG,  suggest  that  at  least  5tJ%  of  members  should 
come  from  an  organization  outside  of  the  SPO.  Provide  an  independent 
input.  Having  members  work  both  on  proposal  evaluation  and  the  PRAG 
created  some  problems.  Whether  the  members  do  not  liave  source 
selection  duties  in  addition  to  PRAG  members  should  be  a  function  of 
the  size  and  complexity  of  the  program.  (57) 

1.  Found  that  there  were  varying  degrees  of  compliance  with  the  CPARS 
requirement  to  include  remarks  on  all  the  ratings.  Need  to  make  sure 
that  the  individual  filling  the  CPAR  out  knows  that  it  will  be  used, 
and  what  it  will  be  used  for.  Need  to  provide  comments  describing 
why  the  area  was  evaluated  as  it  was.  (20) 

5.  Need  to  have  careful  wording  in  the  RFP  to  make  sure  the  instructions 
are  clear  so  that  you  get  the  information  you  really  need.  As  part 
of  the  proposal  have  the  offeror  describe  why  each  contact  is 
relevant  to  the  particular  items.  It  would  help  if  there  were 
standard  instructions  that  could  be  put  into  the  proposal.  (32) 

0.  Make  sure  you  reach  up  in  the  agencies  as  high  as  possible  when 
soliciting  responses  to  the  quest  ionnaires.  Want  somebody’s  name 
besides  the  lowest  level  buyer  on  it.  Would  also  help  to  get  an 
earlier  jump  on  the  process.  (5) 

7.  When  establishing  POCs  to  fill  out  questionnaires  need  to  go  in  at 

t he  senior  management  level  rather  than  the  working  level.  Need  to 
tailor  the  questionnaire  to  the  source  selection.  Have  all  people 
review  past  lessons  learned  before  starting  the  PRAG  process.  (38) 

8.  Use  CR/DR  process  to  handle  concerns.  (15) 

9.  Need  better  policy/operating  instructions  before  you  begin;  spent  a 
lot  of  time  spinning  wheels.  (36) 
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PRAG  Representatives’  Lessons  Learned 

10.  One  lesson  learned  would  be  that  the  questionnaire,  if  used,  should 
be  very  close  to  the  CPAR  format  and  if  possible  simply  using  the 
CPAR  form  would  be  great.  The  other  would  be  to  have  your 
methodology  established  upfront.  (6) 

11.  Important  to  get  the  surveys  out  early.  Get  information  on 
subcontractors.  (25) 

12.  The  names  of  focal  points,  addresses,  and  phone  numbers  provided  by 
the  offeror  were  out  of  date.  Suggest  having  the  offeror  verify  the 
information  before  it  is  put  into  the  past  performance  proposal. 
Second,  need  to  make  sure  that  in  the  RFP  you  instruct  the  offeror  to 
provide  relevant  past  performance  information  on  its  subcont ractors . 
Finally,  during  this  process  they  did  not  have  the  performance 
assessment  completed  prior  to  the  competitive  range  briefing.  If 
past  performance  is  to  be  included  in  the  determination  for  direct 
award  (award  without  discussion)  then  it  needs  to  be  completed 
sooner.  (12) 

12.  Get  questionnaires  out  earlier.  Asked  for  different  informat  ion  in 
the  RFP.  Have  the  offeror  address  each  individual  item.  (10) 

11.  PRAG  members  broken  into  teams,  with  at  least  3-4  members  to  cut  down 
on  the  amount  of  reading.  (15;  23) 

15.  Have  3-ltr  chiefs  perform  assessment  rather  tlian  panel  chiefs.  Panel 
chiefs  were  very  busy  doing  proposal  evaluat ion  tasks,  arid  3-ltr 
chiefs  have  a  larger  experience  base  to  work  with.  (51) 

1G.  Need  to  get  information  from  contractor  early.  PRAG  is  on  the 

critical  path  of  t.he  source  select  ion.  SSA  has  authority  to  award 
with  out  discussions  which  speeds  up  the  process  considerably.  After 
initial  evaluation,  the  competitive  range  is  determined  end  the  SSA 
has  opportunity  to  award  to  best  contractor.  This  initial  evaluation 
is  completed  after  about  5  weeks.  Comparing  this  to  thj  PR\G 
schedule,  the  quest ionnai res  are  just  coming  in.  If  SSA  decides  to 
award  without  discussions,  tne  past  performance  risk  is  never 
completed.  (7;  24) 

17.  Have  been  applying'  lessons  learned.  This  was  the  fourth  program  to 
go  through  the  process.  In  retrospect  more  up  front  work  could  have 
been  done  on  the  questionnaire.  Army,  Navy,  and  Air  Force  just  don’t 
t  hi  nit  alike.  (19) 

18.  Make  sure  quality  of  the  response  to  the  (questionnaire  is  verified. 
Tried  to  send  it  through  a  high  enough  level  (Q-6)  so  that  the 
response  is  reviewed  and  that  it  bears  some  resemblance  to  reality. 
Very  important  that  the  contractor  provides  you  with  the  correct  name 
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PRAG  Representatives'  Lessons  Learned 

that  when  you  are  reviewing-  past  performance  it  is  the  same  division 
that  is  bidding-  on  the  effort  under  consideration.  Make 
sure  that  you  are  reviewing-  contracts  that  are  truly  relevant  to  the 
profit  center/division  that  is  bidding.  Do  a  good  job  up  front  of 
delineating  what  the  item  is.  It  shouldn’t  be  confusing  that  this 
experience  is  relevant  to  item  #1  or  item  #2.  Assessment  points  must 
be  independent  of  each  other.  (44) 


Appendix  E:  Retirements  Base  1  ine 
1 . 8  Data  Col  lection . 

1.1  Process  initiation. 

1.1.1  Interim  and  Final  Assessment  Reports.  The  capability  should  be 
provided  to  track  dates  due  of  interim  and  final  performance  assessment 
reports  in  order  to  give  program  managers  advance  notice.  The  tracking 
capability  should  key  on  either  the  date  of  the  reviewing  official’s 
signature  (Block  21),  the  period  covered  by  report  (Block  3)  or  the 
program  manager’s  organization  (Block  17). 

1.1.2  Initial  Performance  Assessment  Reports.  The  capability  should  be 
provided  to  check  the  Acquisition  Management  Information  System  or  other 
source  for  all  contracts  written  during  a  given  period  vhich  were  for 
concept  definition,  full-scale  development,  or  full-rate  production  and 
cost  five  million  dollars  or  more. 

1.1.2  Hie  capability  should  be  provided  for  the  focal  point  to  keep 
track  of  when  an  assessment  report  is  due,  the  day  that  it  is  due  out  to 
the  contractor,  the  day  it  is  received  by  the  contractor,  the  dav  it  is 
due  back  to  the  government  and  the  day  it  should  be  signed  by  the 
reviewing  official. 

1 . 2  Program  manage  r ’ s  eva 1 ua  t i on . 

1.2.1  The  program  manager  should  be  provided  with  all  information 
necessary  to  complete  the  assessment  report.  Items  should  include  a  copy 
of  the  CPARS  regulation,  local  supplements,  and  any  other  pertinent 
information  or  training  mater ial . 

1.2.2  The  program  manag'er  should  be  presented  the  prior  period’s 
assessment  report.  Descriptive  information  in  Blocks  1  through  13  from 
the  front  of  the  assessment  report  should  be  transferred  from  one  year  to 
the  next.  This  will  ensure  consistency  of  the  information. 

1.2.3  Standard  Language.  Examples  of  standard  language  or  key  words  to 
describe  the  phase  of  the  process,  type  of  technology,  and  type  of  work 
performed  should  be  provided  to  the  program  manager. 

1.2.1  Program  Di rec t or /Manager  Narrative.  Assistance  should  be  provided 
to  the  program  manager  to  ensure  an  adequate  depth  of  information  is 
captured.  Possibilities  should  look  a*  a  list  of  more  detailed 
categories  of  information  for  each  of  the  evaluation  areas  in  Block  1-1, 
and  an  on-line  interactive  environment  in  which  the  program  manager  would 
be  probed  with  questions.  Many  of  the  product  divisions  have  sample 
questionnaires  which  attempt  to  achieve  similar  results  during  interviews 
and  surveys.  Detailed  categories  of  information  identified  by  PR4G 
members  are  provided  in  Appendix  F. 
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1.2.5  The  program  manager  narrative  is  limited  to  the  space  provided  in 
B I ock  16  pi  us  one  add i  t.  i ona I  page . 

1.3  Contractor  Comments 

1.3.1  The  program  direct or /manager  should  be  provided  access  to  sample 
letters  for  transmittal  of  the  assessment  report  to  the  contractor. 
Examples  have  been  developed  by  each  product  division. 

1.3.2  The  CPARS  focal  point  should  have  the  ability  to  review  the 
package  before  it  is  sent  to  the  contractor.  This  is  an  existing 
requirement  for  at  least  one  of  the  product  divisions. 

1.3.3  The  capability  should  be  provided  to  capture  the  contractor’s 
comments  for  insertion  into  the  data  base. 

1.3.4  The  contractor  comments  are  limited  to  the  space  provided  in  block 
lb  plus  one  additional  page. 

1.1  Program  manager  update. 

1.4.1  The  program  manager  reviews  the  assessment  report  and  determines 
if  revisions  are  necessary.  If  revisions  are  necessary  the  informat  inn 
in  Blocks  1  through  5  of  the  assessment  report  should  be  transferred  to  a 
new  form.  The  statement  "Revision  to  CPAR  for  period  mm/dd/yy  to 
mm/dd/yy"  must  appear  in  Block  12.  Blocks  requiring  revision  are  then 
updated  and  the  form  is  appended  to  the  original. 

1.1.2  The  responsible  pa’ty  signs  the  form  in  Block  17  and  transmits  it 
to  the  reviewing  official. 

1.5  Review  Process.  The  capability  should  be  provided  for  the  reviewing 
official  to  make  comments  in  Block  2R  and  sign  in  Block  21. 

2 .  t)  Data  Input . 

2.1  The  capability  muse  be  provide  to  insert  the  assessment  report  into 
the  data  base  at  the  originating  product  division  and  each  of  the  other 
product  divisions. 

2.2  Reports  are  currently  filed  according  to  different  variations  of 
name  of  parent  contractor,  division/subsidiary,  originat ing  product 
division,  and  sequent ia 1 1 y  according  to  contract  number.  At  one  product 
division  a  nine-digit  referencing  system  has  been  established  which 
encompasses  the  name  of  the  parent  contractor,  division/subsidiary,  and 
sequence  of  contract  number. 

2.3  Because  of  limited  resources,  the  system  needs  to  provide  the  most 
simple  procedure  possible  for  input  of  information.  The  goal  should  be 
to  achieve  the  best  mix  of  simplicity,  ease  of  input,  and  increased 
quality  of  information. 
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3 . Q  Main t  enance . 

3.1  The  system  should  nro^  i de  the  capability  required  to  adjust  for 
mergers,  or  contractor  name  changes.  Tht  PRAG  member  needs  to  be  able  to 
ask  for  an  assessment  report  on  a  contractor  and  should  not  have  to  vorry 
what  name  it  is  listed  under  in  the  data  base. 

3.2  Assessment  reports  expire  after  five  years.  The  capability  should 
be  provided  to  search  for  and  remove  expired  reports. 

3.3  The  capability  to  annotate  and  save  assessment  reports  should  be 
provided  to  handle  requests  by  program  managers  to  keep  reports  longer 
than  five  years. 

3.1  Correct  information  on  names ,  addresses,  and  points  of  contact 
should  be  maintained  on  contracts  for  which  a  final  assessment  report  has 
not  been  written.  One  product  division  has  implemented  a  computer  data 
case  of  names  and  addresses  for  those  contractors  which  they  originate 
assessment  reports.  Once  each  year  the  CPARS  focal  point  sends  out  a 
list  to  all  the  contractors  requesting  confirmation  on  the  information. 

4.0  Artalys  is .  The  system  should  provide  capabilities  to  support  the 
performance  risk  assessment  process. 

4.1  PI  arm  i  ng . 

4.1.1  Instructions,  information,  sample  formats,  and  word  processing 
templates  should  be  provided  to  the  program  manager  and  the  PRAG  to  help 
establish  responsibilities  and  a  me+hodology;  make  sure  correct  words  are 
in  the  Request  For  Proposal;  identify  data  needed  to  be  collected  and 
how;  and  determine  in  what  manner  the  data  will  he  analyzed. 

1.1.2  The  system  should  provide  a  sample  format  or  word  processing 
template  to  assist  in  the  preparation  of  the  past  performance  portion  of 
section  M  of  the  Request  For  Proposal,  "Instructions  to  Offeror."  The 
format  or  template  should  remind  the  program  manager  and  PRAG  that  the 
specific  criteria  to  be  looked  at  should  be  identified  and  the  following' 
informat ion  lie  requested  from  the  offeror  for  each  contract:  contract 
number;  acquisition  agency;  progTam  name;  name/address/phone  number  of 
both  the  contractor  officer  and  program  manager;  contract  administration 
office  plus  the  administrative  contracting  officer’s  address  and  phone 
number;  original  contract  cost,  ultimate  contract  cost,  and  reasons  for 
deviation;  original  contract  schedule,  ultimate  contract  schedule,  and 
reasons  for  deviation;  description  of  the  pro jeo*  and  how  that  contract 
was  a  good  (or  had)  example  of  their  ability  to  perform  against  each  of 
the  criteria  stated  in  the  Request  For  Proposal. 

1.1.3  The  capability  should  be'  provided  to  allow  the  PRAG  to  establish 
criteria  for  screening  examples  of  performance  for  relevancy,  and 
determining  the  impact  ot  specific  examples  on  ’he  performance  risk 
assessment.  Criteria  used  in  past  risk  assessment  groups  include  the: 
parent  corporation;  division/profit  center;  t ype  of  report  (initial, 


intermediate,  final);  contract  number;  period  of  performance;  product 
division;  dollar  thresholds;  type  of  contract;  nature  of  the  program; 
end  items;  phase  of  acquisition  cycle;  program  description;  and  uni<^ue 
aspects  within  the  narrative  (quality  assurance,  training,  use  of  special 
software  languages ) . 

4.2  Obtain  Data. 

4.2.1  For  some  time  in  the  future  there  will  be  a  continued  reiiance  on 
surveys  and  interviews  to  supplement  information  which  can  be  extracted 
from  the  CPARS .  The  capability  should  be  provided  to  assist,  the  PRAG 
members  in  the  development  or  tailoring  of  questionnaires.  Attention 
should  be  given  to  wording  of  sample  questions  which  can  be  used  to 
contact  Department  of  Defense  personnel  outside  of  the  Air  Force. 

4.2.2  'fhe  system  should  he  able  to  provide  a  description  of  the  CPARS 
data  base  for  each  contractor.  Suggested  descriptive  capabilities  are 
listed  below. 

4.2.2. 1  Provide  the  capability  to  describe  trends  over  initial, 
intermediate,  and  final  assessment  reports  for  a  selected  contract  or 
across  many  contacts. 

1.2.2. 2  Provide  the  capability  to  describe  for  each  parent  corporation 
and  div is  ion/prof i t  center  a  frequency  of  reports  by  contract  type,  phase 
of  acquisition,  and  program  description. 

1.2. 2. 3  Provide  the  capability  to  describe  for  each  parent  corporation, 
division/profit  confer,  and  contract:  an  average  overall  rat ing  across 
assessment  reports  for  each  evaluation  area;  a  frequency  of  ratings 
across  assessment  reports  for  each  evaluation  area;  and  ii  f re<juency  of 
assessment  reports  within  preset  cost  and  schedule  variance  ranges . 

4.2.3  'The  capability  should  be  provided  for  a  single  office  to  look  at 
the  reports  and  develop  a  corporate  trend  from  the  information  contained 
in  (.PARS.  This  would  then  provid-  PRAGs  across  all  the  product,  divisions 
a  consistent  corporate  profile  for  each  contractor. 

Compile  and  Evaluate  Informat  ion.  Each  PRAG  member  reads  the 
entire'  set  of  forms  to  determine  the  sal  ient  ini  jrmation.  The 
descriptive  blocks  (12  and  13),  eoloi  ratings,  and  narrative  sections  of 
the  (TAR  must  he  used  together  to  develop  the  risk  assessment.  When 
evaluating  .an  offeror,  PRAG  members  check  the  description  of  the  program 
to  see  if  the  informal  ion  was  relevant  and  gauge  the  complexity  of  the 
progr.im.  Members  then  search  the  rat  ings  ;uid  narrative  sections  for 
informal  ion  which  was  relevant  to  the  specific  evaluat  ion  items  mid  areas 
for  their  progrtun.  PRAG  members  use  colors  to  find  trends  in  the 
informal  ion  hut  rely  heav i iv  or.  the  narrat  ive  port  ion  to  subsfanf  iate  the 
rat ings  and  show  relevancy. 

4.3.;  In  order  for  t he  PRAG  representatives  to  determine  if  an 

•  •ssment  report  is  relevant  to  the  pn  gr.-on  under  consideration  the 
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capability  should  be  provided  to  search  through  the  CPARS  based  on 
various  criteria.  Search  capabilities  key  on  information  vhieh  is 
identified  in  standard  fields  and  on  information  which  may  or  may  not  be 
provided  within  certain  fields  on  the  assessment  report.  Examples  of 
criteria  were  listed  in  paragraph  4.1.3.  The  system  should  allow  the 
PRAG  member  to  search  and  sort  assessment  reports;  scan  descriptive 
information  and  color  ratings;  and  focus  on  specific  coirenents  in  the 
program  director/manager  narrative. 

4.3.3  Information  should  be  presented  in  an>  combination  of  the 
following  formats,  a  copy  of  the  original  assessment  report,  a  printed 
copy  in  the  original  format,  some  preset  format  or  a  user  designed 
report.  The  PRAG  member  should  have  the  option  of  working  in  a  paperless 
environment,  if  desired.  Two  examples  of  report  formats  are  presented  in 
Figures  E- i  .and  E-3. 
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Figure  E-l.  Ballistic  Systems  Division  Performance  Data  Matrix 
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Past  Performance  Assessment 


Offeror: 

Product  System  Performance 
Engineering  Design  Support 
Sof t  ware  Deve 1 opment 
S'chedu  le 

Cost  Performance 
Product  Assurance 


Overall  Rating: 

Red  Yellov  Green  Blue 


X 


X 

X 

X 

X 


Figure  E-2.  Munitions  Systems  Division  Contractor  Rating  Sheets 


•1.3.3  A  search  arid  sort  capability  based  on  the  Block  14  color  ratings 
should  be  provided.  This  should  allow  the  PRAG  member  to  search  and  sort 
assessment  reports  based  on  assessment  color  rating  for  each  category,  or 
provide  frequencies  of  ratings  for  each  assessment  category. 

4.4  Perform  Risk  Assessment. 

4.4.1  Performance  Risk  Assessment  is  a  two  step  process  which  first 
cneeks  to  determine  each  programs  relevance  to  the  effort  being 
considered  and  then  subject ively  summarizes  across  all  contracts  so  that 

risk  assessment  for  each  area  could  be  assigned.  Although  not 
specifically  requested  by  any  PRXG  member,  this  process  seems  to  lend 
itself  to  the'  use  of  a  decision  support  system. 

1.4.2  Another  method  used  was  to  show  area  ratings  for  individual 
contracts.  PR\G  members  choose  screening  criteria  and  sort  through 
available  information  to  get  *veral  different  views  of  the  contractor’s 
performance.  The  information  is  display  information  in  a  figures  similar 
t  o  t  hose  sh<  >wn  be  1  ow . 


Summary 


Rat  ings 

Contract  Description  Contract  Dollar 
Number  Of  Effort  Type  Amount  Technical  Management  Cost 


Figure  E-'o.  Ballistic  Systems  Division  Risk  Assessment  Summary 


Remarks 

Technical  Management  Cost 

I  I  I 

I  t  I 

Contract  ]  J  j 

Number  !  !  ! 


Figure  E-4.  Risk  Assessment  Supporting  Information 

o.W  Ou  tpth  .  Output  should  be  available  both  for  display  on  the  computer 
screen  and  in  hard  copy.  The  capability  should  be  available  to:  provide 
ent  ire  copies  of  assessment  reports  which  have  been  determined  to  be 
relevant ;  f^nivide  copies  of  summary  reports  based  on  relevant  fields  of 
the  assessment  report;  provide  output  of  descriptive  statistics  defined 
in  paragraph  1.2.2;  and  download  information  to  the  Performance  Data 
Matrix  (Figure  F- 1 )  and  briefing  charts. 

(i .  t)  Reporting.  Capability  should  be  provided  to  assist  the  CPARS  f  oca  i 
points  in  generating  the  administrative  type  of  reports  described  below. 

tl.l  The  product  divisions  submit  quarterly  or  more  frequently  to  Air 
Force  Systems  Command  a  report  describing  the  latest  assessment  reports 
stat  ing  the  contractor  name,  subsidiary  of  division,  contract  number, 
period  or  performance,  and  a  brief  description  of  the  contract. 
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6.2  Air  Force  Systems  Command  Regulation  800-54,  requires  each  focal 
point  to  submit  annually  a  report  which  states:  the  name  and  address  of 
the  contract  division  or  subsidiary;  the  parent  corporation;  the  number 
of  times  the  contractor  has  submitted  proposals;  and  identify  any  new 
offerors  for  inclusion  in  the  data  base  (1:3). 

6.3  The  focal  point  at  Electronic  Systems  Division  prepares  talking 
papers  stating  the  number  of  assessment  reports  on  file,  the  number  of 
contractors,  number  of  assessment  reports  per  contractor,  and  the  number 
of  programs  which  have  had  PRAC.s  and  looked  at  CPARS. 

6.4  The  focal  point  at  Munitions  Systems  Division  builds  a  report  that 
goes  to  the  commander  on  incomplete  assessment  reports.  Using 
information  gathered  while  tracking  assessment  reports  a  chart  is 
prepared  which  has  a  list  of  contracts,  the  date  assessment  reports  were 
due  out,  the  actual  mailing  date,  the  date  reports  were  returned  by  the 
contractor  and  the  date  of  approval  by  the  reviewing  official. 

7.0  Protect  ion  Of  Informat  ion.  Any  capability  developed  to  support 
CV.XRS  and  the  PRAG  should  consider  the  protection  of  data  since  it  is 
source  selection  sensitive. 

7.1  The  Air  Force  Systems  Command  Form  125  is  marked  "Source  Selection 
Sens i t  i v e-For  Official  Use  Only"  when  filled  out.  Any  output  generated 
by  the  system  should  be  appropriately  marked. 

7.2  When  transmitting  by  mail,  each  assessment  report  is  double  wrapped. 
The  inner  envelop  is  marked  "Source  Selection  Sensitive  -  For  official 
Use  Only".  The  outer  envelope  is  marked  "To  Be  Opened  By  Addressee 
Oily".  The  transmittal  letter  to  the  contractor  highlights  the  fact  t hat 
the  report  is  source  selection  sensitive  and  the  restrictions  on  its  use. 

7.5  Appropriate  control  features  should  make  sure  flint  the  data  cannot 
be  changed  or  retrieved  by  someone  who  should  not  have  access. 

7.4  At  Electronic,  Munitions  and  Space  Systems  Divisions  no  copies  of 
the  assessment  reports  are  made.  The  original  reports  are  given  to  the 
PRAG  members  who  work  with  them  within  the  source  select  ion  facility. 

7.5  The  system  should  he  capable  to  provide  protection  of  t  hi'  data  in 
such  a  way  that  a  PRAG  member  or  contractor  could  be  limited  access  to  a 
specific  contractor.  There  needs  to  be  some  way  to  segregate  the  data 
base  so  that  access  can  be  control  led. 

7.6  As  current  ly  implement  ed  assessment  reports  are  kept  in  a  file 
drawer  in  a  locked  room  or  vault  with  controlled  access  arid  are  only 
released  to  PRAG  members  with  authorized  access.  Any  capability 
developed  should  not.  degrade  this  level  of  protect  ion. 

7.7  Whil*'  in  transit  between  offices  the  assessment  report  is-  har J 
carried,  kept  covered  with  a  source  selection  sensitive  cover  sheet  and 
is  never  left  alone. 
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Appendix  K:  Information  Types . 


Engineering  Design/Support 


Present  alternative  solutions 
Engineering  Change  Proposal 
processing  t  ime 
Requi remen ts  .Analysis 
Systems  Engineer i rig 
Technical  analysis 

Product 


Correction  of  deficiencies 
Initiate  solutions  to  problems 
Level  of  competence 
System  Design 
System  Integration 


Assurance 


Environmental  Performance  -  Installed  System 

Performance  -  Quality  Engineering 

totality  of  product 


Software  Development 


Computer  Resources 


Management  Responsiveness 

Flexibility  Initiative  -  Proactive  approach 

Prompt  resolution  of  issues 


Schedu 1 e 


Mast  or  Integrated  Schedule  -  Meeting  Major  Milestones 

Subcontractor  Management 


Competent  resources 


Integrated  Logistics  Support 


Fac i 1 i t i es 

Ma  i  ri  t  enance  P 1  /inn  i  r  ig 
Support 
P 1  ann i ng 


Implement  at  ion 
Spares  Avai labi 1 i ty 
Support  Requi remen ts 
Technical  Data 


Cost  Performance 


Cos  t 

Cost  Management 
History  of  Buy-in’s 


Cost  control 

Elements  of  cost  proport ional 
to  targets 


F-  1 


Other  Categories 


four  s  e  deve  l  opmert  t 
Instruction  System  Deve 1 opmen t 
Plan  for  Reliability, 
Maintainability  and 
Producibi 1 ity 
Training  System 
User  involvement 
Con  figuration  ,/Da  t  a  Managemen  t 
Focus  on  key  problem  areas 
Organizational  Control 
Overall  planning  and  management 
Risk  Managemen  t 
Technical  Management  PI  tunning 
Top  Management  Participation 
Manu  f  ac  t  ur i ng  Produc i b i i ity 
quality  Assurance  Program 
War r an  t  y  Managemen  t 
Field  Availability  Rates 
Logistics  Support.  .Analysis 
Support ab i 1 i t  y 
Show  cause  or  stop  uork 
act  ivit ies 

Engineering  change  proposals 
veil  prepared 


Ineorporat ion  of  commercial  off 
the  shelf  and  non  development 
items  into  the  overall  design 
Prototype  Management 
Training  Delivery 
Concept  Definition 
Acquisition  Support 
Engineering  Change  Proposal 
Process 

Organizational  Resources 
Quality  of  Contract  Data 
Requirement  List  deliveries 
Timely  Contract  Data 
Requirement  List  delivery 
Program  Management 
Second  Sourcing 
Work  Measurement 
Life  Cycle  Costs 
Service  Reports  and  Material 
In  Processing  Review  Boards 
Use  a  reasonable  and  rational 
budge  t 


Appendix  G:  Description  of  initial  Automated  Capability 


Overview.  Actual  development,  test  and  implementation  of  the 
operational  CPARS  automated  information  system  will  take  some  time  and 
significant  effort.  At  the  same  time,  the  ability  for  DRAG  members  to 
determine  which  assessment  reports  are  relevant  and  the  need  for 
simplified  CPARS  administration  should  be  met  as  soon  as  possible.  For 
t  best'  reasons  an  interim  capability  was  developed  using'  the  integrated 
software  package  Enable  (Version  2.15)  find  an  IBM  XT  compatible  computer 
sysf cm. 

liable  was  used  to  create  a  data  base,  input  forms,  and  several 
report  forms  which  combine  the  basic  functions  of  the  automated 
capabilities  at  both  Ballistic  Systems  Division  and  Aeronautical  Systems 
Division.  Functions  were  designed  to  allow  for  flexibility  and  growth, 
considering  that  implementation  may  he  dependent  on  the  eventual 
capabilities  of  this  system. 

Setting  up  the  system  will  require  some  prior  knowledge  of  friable  or 
at  least  the  wi 1  I ingness  to  1  earn.  Having  the  Get t  j ng  Started  volume  of 
the  Enable  users  manual  may  he  1 p . 

Enable.  Enable  is  on  integrated  software  package  containing  five 
different  applications:  word  processing,  data  base  management, 
spreadsheet,  graphics,  and  telecommunications  (22:Sec  1,1).  The  friable 
data  base  managemen t  application  irt  combination  with  the  word  processing 
functions  provide  t  he  necessary  capability  to  complete,  store,  search, 
print,  find  maintain  CPARS  information,  finable’s  report  generating 
funet ions  provide  sufficient  tools  to  create  complex  reports  such  as  the 
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Air  Force  Systems  Command  Form  125,  and  is  simple  enough  to  quickly 
develop  "user  defined"  reports  as  needed.  By  taking  advantage  of 
Ehable' s  menu  driven  environment  and  through  the  use  of  macros  and 
windows,  a  tolerant  atmosphere  could  he  developed  to  put  at  ease  even  the 
most  novice  of  computer  users. 

Base  Organ izat  i on.  'Ihe  primary  data  base  for  the  automated 
capability  was  designed  to  simplify  data  input  and  to  allow  as  much 
flexibility  for  search/sort  routines  as  possible.  A  total  of  124 
different  fields  were  used  to  capture'  and  store  the  information  required 
to  complete  the  front  page  of  the  CPAR  form.  The  data  base  definitions 
are  contained  in  the  data  base  definition  file  called  CPARS.DBF. 

Stating  that  a  primary  data  base  was  developed  is  a  bit  of  a 
misnomer.  in  reality  four  different  data  bases  are  used  to  accumulate 
the  final  assessment  reports  which  appear  in  the  CPARS  delta  base.  The 
data  bases  and  their  relationships  to  each  other  are  shown  in  Figure  G- 1 . 

The  NAMES. DBF  data  base  contains  the  parent  corporation  name, 
di\ is  inn/subs i diary ,  address,  Department  of  Defense  Activity  Address 
Directory  Code  (DODAAD),  and  contractor  point  of  contact.  Each 
combination  of  parent  corporation  and  division/subsidiary  is  represented 
by  a  single  record.  No  duplication  of  division/subsidiary  names  are 
accepted.  Information  resident  in  this  data  base  is  independent  of  the 
specific  contract,  product  division,  or  type  of  assessment 
( i m i t i a  I / i  rd  ermed i at  e/f i na 1 ) .  This  data  base  is  similar  to  the 
name/address  data  base  discussed  earlier  in  Chapter  5.  A  field  for 
contractor  point  of  contact,  at  the  division/subsidiary  level  was  added  to 
the  data  base  definition  to  support  maintenance  functions.  Because  the 
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NAMES . DBF 


Parent  Corporation 
D  i  v  i  s  i on/Subs i d i ary 
Address 

City  State  Zip 
DODAAD 


V _ ASSESSMENT .  DBF _ 

Fields  included  in 
CONTRACT.  DBF 
+ 

-nit.  ial/lntermediate/Final 
Period  Covered  by  Report 
Contract  Percent  Complete 
Current  Dollar  Value 
Area  Asses smen t s  (Block  14) 
Cost/Schedule  Variance 


V  CPARS . DBF 


Includes  all 
assessment  reports 
and  revisions  to 
assessment  reports 


CONTRACT.  DBF 


Fields  included  in 
NAMES. DBF 
+ 

Contract  Number 
Location  of  Performance 
Contract  Period  of  Performance 
Type  of  Competition 
Contract  Types 

Title  and  Phase  of  Acquisition 
Contract  Effor^  Description 
Product  Division 


_ REVISED . DBF 

Fields  included  in 
NAMES. DBF 

T 

In it ial/lntermediate/Final 

Period  Covered  by  Report 

Contract  Number 

Product  Division 

Notice  in  Block  12 

Rev i s i ons  t  o  B 1 ocks  14,  15 


Figure  G-  1 :  CPARS  Automated  Information  System  Data  liases. 

name/address  informat  ion  is  contained  in  an  independent  data  base, 
informal  ion  can  be  easily  updated  without  altering'  any  existing  validated 
assessment  report s . 

The  CONTRACT . DBF  data  base  contains  information  from  the 
name/address  data  base  as  well  as  informat  ion  pertaining  to  a  specific 
contract  .  Fields  included  in  this  data  base  are  contract  dependent,  but 
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independent  of  the  type  of  assessment.  Information  is  retrieved  from  the 
name/address  data  base  as  new  contracts  are  added  to  CONTRACT . DBF .  Each 
contract  number  is  represented  by  a  single  record.  No  duplication  of 
contract  numbers  are  accepted. 

The  ASSESS. DBF  data  base  contains  the  information  elements  relevant 
to  the  actual  assessment  for  a  given  year.  The  information  contained  in 
this  data  base  is  dependent  both  on  contract  and  on  the  actual  assessment 

from  the  prior  period.  The  assessment  data  base  is  used  for  a  storage 

mechanism  until  a  given  assessment  is  validated. 

The  REVISED. DBF  is  used  to  develop  and  temporarily  store  a  revision 
to  a  preliminary  assessment.  The  revision  data  base  contains  information 
elements  from  blocks  1  through  5  of  the  Air  Force  Systems  Command  Fora 
125,  as  well  as  a  message  in  Block  12  stating  that  it  is  a  revised 
report.  Adjustments  are  made  in  Blocks  14  and  15,  and  a  record  is  saved 
for  later  transmission  to  the  CPAES  data  base. 

After  the  assessment  report  and  the  revisions  are  reviewed  and 
validated  by  the  product  division  reviewing  official,  they  are  copied  to 
the  (PARS  data  base  for  permanent  storage.  The  data  bases  are  set  up  so 
that  no  changes  can  be  made  to  an  assessment  report  on°e  it  is  entered 

into  the  (PARS  data  base.  Ibis  was  accomplished  by  specifying  for  each 

field  in  the  (.PARS  input  form  that  no  update  was  allowed.  When  an 
assessment  report  is  transferred  to  the  (PARS  data  base  for  permanent 
storage,  the  corresponding  records  in  !  tie  assessment  and  revision  data 
liases  are  removed  and  destroyed. 

Gettjng  Started ■  The  (PARS  Aut  omated  Informat  ion  System  disk  should 
he  copied  to  a  directory  named  C:\CPARS  prior  to  operation.  The  system 
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can  be  operated  from  a  floppy  disk  drive  but  it.  is  not  recommended  due  t.o 
the  slower  responsiveness,  arid  the  eventual  storage  problems  which  will 
occur.  finable  Version  2.15  should  be  installed  in  a  directory  named 
C:\ENABLE.  The  CPARS  automated  system  is  started  by  typing  "CPARS"  at 
the  C:\CPARS  prompt.  The  instructions  contained  in  the  CPARS. RAT  file 
will  initialize  Enable  and  execute  a  macro  which  presents  the  CPARS 
Automated  Information  System  (A1S)  Main  Menu. 

Enable  uses  profiles  to  establish  the  default  settings  for 
operation.  Default  values  can  be  set  for  printers,  plotters,  page 
format,  screen  format,  word  processing,  etc.  The  default  settings  for 
the  CPARS  A  IS  are  included  in  the  profile  named  CPARS.  The  CPARS  profile 
i1--  automatically  executed  with  the  start  up  procedure. 

Any  of  the  default  values  in  this  file  can  be  changed  except  for  the 
default  ruler.  The  ruler  options  provide  an  automatic  left  and  right 
margin  for  all  documents.  The  current  settings  in  the  CPARS  profile  have 
the  default  ruler  off.  This  should  remain  off  to  protect,  the  format  of 
exist  ing  reports.  Nearly'  every  output  form  generated  by  the  system  is 
greater  than  80  characters  wide.  A  default  value  1*  ”  nan  that,  will 
destroy  the  format  of  these  reports. 

Two  default  values  which  the  user  may  want  to  change  are  the  printer 
being  used  rind  the  type  of  monitor.  Current  setting's  for  these  are  an 
uni  deni  ified  printer  and  an  EGA  compatible  color  monitor. 

iterating  Procedures.  Ins  ructions  for  successful  operation  of  the 
system  are  listed  at  the  bottom  of  the  CPARS  Automated  Information  System 
menu.  Pressing  [ESC)  will  exit  the  A IS  .Main  Menu  and  inter  Enable. 
Pressing  [Shift./F1M|  will  cause  the  system  to  return  to  the  AIS  Main 
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Menu.  After  selecting’  an  applicat  ion  from  a  menu,  and  unless  otherwise 
instructed,  the  user  should  follow  the  Enable  prompts  in  the  status  line 
at  the  bottom  of  the  screen.  In  most  cases  the  system  will  automatically 
return  the  user  to  the  menu  from  which  the  option  was  selected.  For  some 
applications  the  user  is  required  to  follow  the  system  prompts  to  return 
to  the  Enable  system  menu.  The  user  can  then  press  [SHIFT/F10J  to  return 
to  the  CPARS  AIS  Main  Menu. 

There  are  two  methods  for  executing  options  on  any  of  the  system 
menus.  The  first  method,  using  the  up  and  down  arrow  keys,  should  be 
practiced  until  familiarity  of  the  system  is  established.  Options  are 
highlighted  when  touched  by  the  cursor.  Many  of  the  options  have 
information  windows  which  further  describe  the  function.  Once 
highlighted,  options  are  executed  by  pressing  the  [Ehter]  key.  The  more 
familiar  user  of  the  system  can  execute  applications  with  a  single 
keystroke  by  entering  the  number  which  directly  proceeds  the  option  on 
the  menu. 
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